The SF 3 M approach to 3-D photo-reconstruction for non-expert users : application to a gully network

Introduction Conclusions References Tables Figures


Introduction
3-D photo-reconstruction (PR) based on structure-from-motion (SfM) algorithms has been applied to date to a large number of geoscience applications (James and Robson, 2012;Westoby et al., 2012;Fonstad et al., 2013).Although there has been a great advance in the last years regarding imagery collection (for instance, derived from the development of UAV platforms) and image processing (commercial as well as free software), the complete photo-reconstruction (PR) and analysis workflow frequently Introduction

Conclusions References
Tables Figures

Back Close
Full remains lengthy and not straightforward (Kaiser et al., 2014).If using freely available software, it requires working on a number of different applications to cover basic image pre-processing, photo-reconstruction, georeferencing and post-processing operations.
Commercial PR software generally has the ability to perform full PR workflows, but can lack detailed processing information and can restrict user interaction with intermediate and final results.
While recent UAV technologies have the capacity of surveying large areas of the landscape (Mathews and Jensen, 2013;Mancini et al., 2013), not all stakeholders (e.g.researchers, technicians and land owners) have the technical and financial resources to use such sophisticated techniques.In addition, government regulations in several countries are becoming increasingly stringent for UAV operations, which hampers the widespread application of this tool.
Thus, there is still a need in developed and developing countries to implement efficient terrestrial PR methodologies (in terms of budget and time requirements) for scientific and technical users concerned with geomorphological processes, such as gully erosion.Fonstad et al. (2013) suggested that terrestrial PR techniques could improve their cost-benefit performance by using multiple operators, poles, video capture or a combination of terrestrial and aerial images.New technologies (such as light-weight cameras) and recent computer developments offer new opportunities to improve data collection and processing.Powerful and freely available software applications have been developed -e.g.VisualSFM for photo-reconstruction (Wu, 2013) or CloudCompare for cloud processing (Girardeau-Monteaut, 2015) among others -and are being constantly improved through the valuable effort of their developers and users' feedback.
3-D PR has been used for gully erosion assessment at the gully reach or headcut scale (Castillo et al., 2012a;Kaiser et al., 2014;Gómez Gutiérrez et al., 2014) and ephemeral gullies (Castillo et al., 2014a), usually not more than over a few meters extent.However, the fully characterization of gully erosion requires the assessment of entire gully networks to understand their geometry and dynamics and this brings several challenges for terrestrial PR: (1) morphological complexity: gullies comprise Introduction

Conclusions References
Tables Figures

Back Close
Full (2) valley location: gullies are deep trenches that do not facilitate easy all round image collection, hampering multiple convergent perspectives; (3) linearity: gullies present very high length/width ratios making the photo-reconstruction models more vulnerable to systematic errors.
As an effort to facilitate the use of freely available PR software for demanding gully erosion applications, here we develop a combination of a rapid survey method and SF3M, a workflow software tool for efficient processing of accurate 3-D models of gully networks at a reduced cost.For this purpose, (1) we present SF3M, a new graphical user interface to guide PR workflow carried out with existing freely available software; (2) we describe a field methodology for the rapid assessment of gully networks; and (3) we evaluate their performance and the 3-D model accuracy with a study case of gully erosion in the Campiña landscape.

Material and methods
2.1 SF3M: a GUI for efficient photo-reconstruction SF3M v1.0 has been devised as a freely available tool for semi-automated 3-D PR to offer a complete workflow from the image set to the 3-D model.SF3M v1.0 is written in Matlab ® (Mathworks, Natick, MA, USA) and comprises algorithms developed by the authors of this manuscript, a number of previous scripts written by other authors (Table 1) as well as external calls to free software such as VisualSFM (Wu, 2013(Wu, , 2015) ) and CloudCompare (Girardeau-Montaut, 2015).SF3M takes advantage of the command line possibilities already present in these external applications to perform key operations, such as photo-reconstruction (including SIFT features detection, bundle adjustment, sparse and dense reconstruction inside the VisualSFM package) and point cloud processing (i.e. point density, filtering and merging operations in CloudCompare).
The GUI is organized in three windows: main, image and display window (Fig. 1).
The main window allows the user to define the operations to be performed.The image Figures

Back Close
Full window can be used to visualize a photograph, and to enter and delete ground control point (GCP) observations.The display window gives information on the stage in process, time left to finish and main results.SF3M v1.0 use follows a sequential process including pre-prepocessing, reconstruction, georeferencing and post-processing stages.All the command options are displayed in the main window so that the user can enter the processing options in advance and leave the application running automatically.This design is intended to keep the GUI operation as simple as possible, facilitating its use for non-trained users.
The main features of the SF3M are outlined in Table 1.For a more detailed description of SF3M v1.0 functionalities, the SF3M executable and instructions are available at http://sf3mapp.csic.es.

Field methodology for rapid gully network assessment
We designed a methodology of field image collection for rapid gully erosion reconstruction based on four principles: 1. automated image collection from a pole to capture high centre-perspectives of the gully from its perimeter; 2. simultaneous capture of two perspectives (two cameras needed in the pole), with one vertical and the other inclined, to: (1) maximise the probability of successful image matching between photographs taken from different sides of the gully, in order to achieve a single 3-D model; (2) ensure a convergent imaging geometry to help minimise systematic reconstruction errors and model distortion; 3. image capture from only one walking itinerary along the entire gully perimeter; 4. use of low-cost devices and materials for image collection.
As an application of the SF3M processing method, a gully erosion survey was conducted in a gully network close to the city of Córdoba (Spain) covered with field crops Introduction

Conclusions References
Tables Figures
The gully was 510 m long in its main channel and as 11 m wide and 3 m deep in its larger cross sections.Each of the three gully branches were taken as separate photo-reconstruction units in the field and processing stages.The gully network was selected for its convenient location near to Córdoba city as well as for having been filled in during 2008, which provides a baseline to estimate recent gully erosion.
In this study, two GoPro Hero3+ cameras equipped with non-fisheye lenses were mounted on a 6 m-long pole (made from a 9 m-long carbon-fibre telescopic fishing rod) to gain height to capture the gully dimensions from a centre position in time lapse capture mode (1 s interval).To reduce camera vibrations, the unused three meters of the telescopic 9 m-long fishing rod were secured with plastic cable ties at the end of the 6 m pole.Both cameras were fixed to the pole end and held in a horizontal position with the help of plastic wedges and cable ties.One camera looked down in a roughly nadir perspective and the other was inclined to around 10 • .This camera arrangement was intended to facilitate the image matching between pictures taken from different sides of the gully along the perimeter itinerary in order to obtain one single 3-D model with geometric consistency (Fig. 3a and b).
An uninterrupted itinerary along the gully perimeter was followed at a slow walking speed, starting from one point and ending on the opposite side of each gully branch (Fig. 3c).Forty five targets, hereafter called ground control points (GCPs), of 20×20 cm dimensions in two colours (pink for even numbers and yellow for odd numbers) were deployed next to the gully edges and at the gully bottom and measured with differential GPS (dGPS, 2 cm accuracy) for georeferencing and error evaluation purposes.

Processing methodology in SF3M
The key features of the processing workflow followed using SF3M v1.0 are listed below: 1. Simplicity in the design and operation, with only one GUI window to define the array of algorithms to run.The typical operation for a project reconstruc-Introduction

Conclusions References
Tables Figures

Back Close
Full tion would include five sequential steps: (1) pre-processing to prepare the image set (automated); (2) project definition, to check the image connectivity and the number of subprojects to process (semi-automated); (3) photo-reconstruction (automated); ( 4) georeferencing (semi-automated); (5) post-processing (semiautomated).Within each of these steps for a PR project, several options can be selected for successive processing, e.g.blurry images detection + image undistortion or green index filter + density filter + Merge + DEM+ SfM accuracy.In addition, if the batch mode is activated, the selected operations can be performed over different PR projects.
2. Camera calibration: SF3M does not perform camera calibration.However, SF3M settings include the option to enter camera parameters (focal length and principal point) for use with undistorted images and the fixed camera calibration mode in VisualSFM.In this work, we could not take advantage of this option because VisualSFM does not allow simultaneous use of more than one camera in this mode.Therefore, the internal camera parameters were estimated automatically during the bundle adjustment in VisualSFM.
Furthermore, SF3M allows the user to undistort images if camera calibration data is available from the Matlab Calibration Toolbox (Bouguet, 2014).For our study, both cameras were calibrated using this toolbox and all images were then undistorted keeping the VisualSFM "determine radial distortion option" disabled.
3. Pair preselection: a first fast run of VisualSFM is performed to identify the significant matches between image pairs (pairs with inlier matches) by comparing all the possible pair combinations, but using only a relatively small number of features from each image for speed.SF3M sets a limit of 1200 features to use in the VisualSFM settings (the default value is 8192) for the project connectivity analysis.This value has proved to be sufficiently large to accurately capture the image connection, but at a minimum processing cost.Introduction

Conclusions References
Tables Figures

Back Close
Full As a result of this analysis, a list of connected image pairs is generated, which will be the input to the final matching stage.This list includes generally much fewer image pairs that all the possible combinations among the pictures.The number of the possible combinations (and consequently, processing time) follows a square power law with the number of pictures, while normally the image connection is highly linear (only pictures in the neighbourhood share common features).This results in a significant reduction on the match processing duration, one of the more time consuming stages.
4. Subproject delineation: in SF3M the project connectivity analysis not only checks for the production of a single model and generates an optimal pair list but also provides the approximate locations of cameras with GCPs (provided that GCP observations have been entered by the user in the image window and the search GCPs algorithm performed).This option allows the delineation of subprojects, which are reconstructed separately, by selecting the cameras to be included in the analysis with a polygon drawing tool.Overlapping areas between subprojects are recommended to reduce the errors in the merging process.This tool is advantageous for field surveys where images of the same areas are taken at different times (e.g. in a gully, the upstream and downstream walks on different sides of the same gully region) and to minimise the effects of systematic errors in large image sets.
5. Photo-reconstruction: in SF3M v1.0 the photo-reconstruction stage is carried out through a system call to VisualSFM using command line syntax to drive automated processing.The main VisualSFM commands used in our utility are: extracting SIFT features, image matching, exporting match matrix, bundle adjustment and dense reconstruction (multi-view stereo PMVS2 software, Furukawa and Ponce, 2010).
6. Georeferencing: we followed a similar approach to that developed in SfM_georef (James and Robson, 2012).The GCP observations are entered manually in the Introduction

Conclusions References
Tables Figures

Back Close
Full image window.The GCP table in the main window provides information on the number of observations per GCP, the image errors (in pixels) and the absolute errors (m).The tool gives the option of selecting which GCP are to be used for georeferencing (georef GCPs) and which other are used to evaluate errors (control GCPs).
7. Dense cloud filtering: SF3M includes two optional filters to be applied to the dense point clouds.The green index filter removes those points with a green index above the threshold selected by the user, for instance, green parts of vegetation standing at the banks or gully bottom.The green index is calculated using GI = 2g − b − r, where r, b an g stands for the pixel value of each of the colour bands in the RGB image (Meyer and Neto, 2008).The density filter is intended to remove those points with a low point density in their neighbourhood, typically related to lower accuracies.It is also helpful to remove inaccurate points in overlapping areas shared by two dense point clouds which may reduce the accuracy of pre-existent more accurate points.The density filter is performed automatically through command line calls to two CloudCompare algorithms: density calculation and filter by point value.To filter non-green vegetation (long-standing greyish prickly weeds, in our case) the point classification algorithm CANUPO (Brodu and Lague, 2012) was used, through its inclusion as a ready-to-use script in CloudCompare.CANUPO performs a point classification into two groups (in this example, weeds and soil) after a training stage carried out by the user.
10. Results: SF3M provides the DEM (average of the z values of points included in a cell of a size defined by the user), a point density map (points m −2 ) and an "SfM precision" map.A decimation factor can be included to speed up the processing, i.e. a factor of ten would include only a tenth of the points in a cell for the computations, for processing large or particularly dense point clouds.We use the term

Field method performance
Table 2 shows the field and processing time requirements for this study.A total of 6650 images were taken in the field for the entire gully network at a rate of one picture per second and camera.Approximately 90 min of effective labour (travel to study area, pole preparation and GPS base stationing not considered) were necessary for the field survey.Image collection, and GCP deployment and measurement both had similar time requirements, ∼ 90 min each.Two operators participated in the survey, one for the image capture and the other for GCP measurement, working simultaneously for efficiency purposes.
If no georeferencing had been necessary (for instance, if there is no need of several time series comparison) a much faster approach for scaling and orientation might be followed, using levelled objects of known size, as in previous works (Castillo et al., 2014b;Kaiser et al., 2014).In highly linear models such as the gully network, simple procedures (e.g. a carefully levelled several-meters-long thin rope commonly used in construction works) are applicable for later scaling and orientation using a point cloud editing software.
Over recently ploughed gully margins, a walking speed of ∼ 1.5 km h −1 (approximately a third of normal speed) was necessary to avoid undesired movement in the camera and blurred images.Despite the lightness of the selected materials (pole and cameras) a short break in the image collection was made after completing each of the gully branches to avoid operator fatigue.The camera height (∼ 5 m for the inclined 6 m-long pole) was enough to capture the gully dimensions in its larger cross section (11 m).This gully width seems the maximum achievable for the present image collection design.Introduction

Conclusions References
Tables Figures

Back Close
Full

SF3M processing performance
With images taken at 1 s intervals, the image set could be reduced by a factor of two due to repetition within the pictures.The remaining 3275 images were automatically analysed for blur and 180 images with a blur index greater than two standard deviations above the mean value, were discarded (Fig. 5a).
As expected, the project connectivity showed a multiple-matching pattern reflecting the nadir and tilted camera angles and downstream and upstream walking directions (Fig. 5b).The match matrix diagonal corresponds to the connection between consecutive images; additional lines parallel or perpendicular to the diagonal reflect connectivity between upstream and downstream image groups from the same or different cameras.
The gully network was processed as three different PR projects (reflecting the number of gully branches) which were then merged in a single point cloud.A total of 2960 min were necessary to process the entire gully network, i.e. ∼ 49 h, of which 17 % required operator assistance and 83 % only computer time.Photo-reconstruction (54.5 % of the total time) and picture undistortion (15.2 %) were the more timeconsuming stages.
All the processing steps were performed using SF3M v1.0 except for those specified in the methods section which required point cloud editing in CloudCompare: (1) subproject merging for the chunks reconstructed separately inside a gully branch to reduce model deformation; (2) project merging for the three gully branches; (3) non-green vegetation filter using CANUPO.Regarding the vegetation filtering, firstly the automated green index filter was applied inside SF3M resulting in a 3.5 % of the total points in the model removed corresponding to small green weeds (Fig. 6a).The second main type of vegetation in the gully was a tall-standing greyish weed, which was filtered using CANUPO.The points classified and removed as this type of vegetation amounted to 4.1 % of the total, although its detrimental effects on the model were significant at some gully bottoms areas (Fig. 6b).

Conclusions References
Tables Figures

Back Close
Full The final gully model had > 17 million points over a plan area of 4230 m 2 , making a point density average of 4020 points m −2 (Fig. 6c), or approximately 1 point every 2 cm 2 , which was sufficient for the 3-D modelling purposes at this scale.A higher point density (roughly twice the number of points) could have been achieved by selecting the maximum density level in the pmvs2 algorithm at the cost of longer processing time in the dense reconstruction stage (approximately by a factor of 5).In many applications, higher densities might be impractical and not required for DEM construction.The final photo-reconstruction percentage (the cells with elevation values to total cells ratio) for the 0.25 m DEM was 74.25 %, due to the abundant vegetation occlusion at the gully bottom in certain areas.

3-D model accuracy
For the present application, an average GCP error of 6.9 cm was found (Fig. 7a).The GCP error is defined as the distance in world coordinates between each GCP centre location in the final georeferenced point cloud and the GCP centre coordinates measured by dGPS in the field.When compared with the average local SfM precision value (Eq.1), the average GCP error was significantly larger, i.e. 6.9 cm against 2.5 cm (Fig. 7b).
There are a variety of sources of error as a consequence of the inexpensive and rapid survey methodology selected in this study such as low-quality camera lenses, uncertainty in the internal camera parameters of the GoPro camera models, reduced number of perspectives from images with only two main angles and the low number of pictures per spatial unit.All these factors may contribute to error in the 3-D model both on the local scale (i.e. in the form of uncertainties in the point position) and the model scale (geometrical deformations due to systematic errors accumulating over the model extent).
The discrepancy between the GCP error and the estimated SfM precision can be explained mainly as the result of the model deformation at the several-tens-of-meter Introduction

Conclusions References
Tables Figures

Back Close
Full scale, and is likely to reflect residual error in the camera calibrations (James and Robson, 2014).This deformation is visualized as an apparent dome effect at both extremes of each dense point cloud, with increasing error for larger distances from the point cloud centroid.Although the image collection was designed to minimise this effect by specifically including inclined images, doming effects remain noticeable.
In this study, the most successful strategy to mitigate the model deformation was to divide the photo-reconstruction project in different subprojects for separate reconstruction using the subproject tool in SF3M.The length of each subproject was defined by the condition of including at least 4 GCPs.Since a total of 45 GCPs were deployed in the field, an average length of ∼ 50 m per subproject was obtained.This approach was advantageous in terms of overall accuracy but implied an additional processing time for manually stitching and merging the 17 subprojects in CloudCompare by determining the area of minimal error between adjacent point clouds.
If the subproject definition strategy would not be applied, for instance by reconstructing a several-hundred-meters gully reach in one project, the deformation errors would have been in the order of several tens of centimeters.Similarly, James and Robson (2014) found doming deformations of ∼ 0.2 m over horizontal distances of ∼ 100 m for simulations representative of UAV flights.Their recommendations on using fixed calibration to avoid doming errors could not be followed because VisualSFM does not include a fixed calibration option for multiple cameras.
When compared with previous studies on gully erosion assessment through SfM photo-reconstruction (Castillo et al., 2012a;Gómez-Gutiérrez et al., 2014;Kaiser et al., 2014;Castillo et al., 2014) in terms of unitary efficiency (field effort per meter of gully), this work showed larger errors (roughly 2 times the average errors in those studies) but at a lower survey intensity (images per meter of gully) and at one order of magnitude lower time requirements (Table 3).Introduction

Conclusions References
Tables Figures

Back Close
Full

Gully erosion estimate
The resulting DEM was used to estimate the gully volume and a gully erosion estimate, taking the 2008 filled situation as a reference.The volume was determined using the Cut and Fill algorithms in ArcGis ™ 9.3 (ESRI Inc., Redlands, CA, USA).The gully limits were delineated manually by interpreting the DEM and slope maps since, in some areas, the gully rims were not well represented in the dense point cloud and automated methods were not fully applicable.A total gully volume of 3484 m 3 was obtained for a drainage area of 10.9 has at the gully network outlet.We assumed that the gully was filled in the summer of 2008 (common period for this operation in the Campiña landscape) since the orthophotography (April 2009) shows the gully already filled (Fig. 8).Also, due to the similarity in the gully network of 2011 and 2014 (present study), there is no evidence of further major filling operations between these dates.
Considering a bulk soil density of 1.5 Mg m −3 , typical of vertic soils in our conditions and a time span of six years, an average gully erosion estimate of 79.5 Mg ha year was calculated.Most likely, the peak of gully erosion took place during the 2009 and 2010, a wet period with annual rainfalls exceeding 1000 mm in the area preceded and followed by seasons closer to the average (650 mm per year).This high value of mean gully erosion is in agreement with previous assessments for similar conditions over the same period (Castillo et al., 2012b).

Conclusions
3-D photo-reconstruction techniques based on SfM algorithms have already demonstrated their capability for producing accurate 3-D models in a range of geoscience applications.Nevertheless, research is still needed to improve efficiency in a number of challenging situations and their ease of use for workers not necessarily skilled in photogrammetric applications.SF3M v1.0 proved to be an efficient and flexible tool Introduction

Conclusions References
Tables Figures

Back Close
Full for 3-D photo-reconstruction in these regards, considering its simplicity and complete workflow.
To the author's knowledge, this is the first time an entire several-hundred-meters-long gully network has been surveyed by terrestrial photo-reconstruction.This was carried out using inexpensive means (around EUR 1000 budget for the field equipment), little manpower (a minimum of one operator is required), in a short time span and has achieved moderate accuracies.Therefore, the survey design and processing methodology included in this study is a promising tool for gully erosion evaluation in scenarios with demanding budget and time constraints and reduced operator expertise.Introduction

Conclusions References
Tables Figures

Back Close
Full    Full  Full  de Andalucía, 2015).Most probably, the gully was landfilled in the summer of 2008 and, since then, there is no evidence of having been filled again.
Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | long networks of varying size along their length; Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Screen / Esc Printer-friendly Version Interactive Discussion Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | 8. Merging dense point clouds: SF3M automatically merges all the dense point clouds belonging to a subproject using CloudCompare in command line.If subproject definition is not carried out, the resulting merged file is the final dense point cloud.For multiple subprojects, to obtain the final point cloud, the intermediate subproject clouds need to be merged manually by the user (step 9).9. Point cloud editing: two manual operations were performed fully inside CloudCompare: subproject and project merging, and non-green vegetation filtering.The merging procedure involves the following algorithms: (a) distance cloud-tocloud of adjacent clouds to determine the specific area inside the overlapping region with minimal errors; (b) cropping both point clouds along this area of minimal error to remove the overlapping; (c) merging the point clouds.Discussion Paper | Discussion Paper | Discussion Paper |

"
SfM precision" to describe local uncertainties in the sparse point cloud.For each point, the local uncertainty is approximated by considering (a) the maximum image error, calculated as the distance between the image measurement and the point projection for the image with the largest error; (b) image focal length (c) the camera-to-point distance in world coordinates.This SfM precision is estimated with the following expression: local precision is higher than the final accuracy measured by ground control reference, since it does not include any wider geometric distortion that may exist across a model.Nevertheless, it provides a useful quantification of photoreconstruction error on a local basis.Finally, SF3M v1.0 saves relevant outputs in the main folder for further edition by the user in csv, txt or ascii formats such as matching features, transformation matrices, DEM or point density map, among others.Figure4shows the SF3M workflow with indication of the command options to be selected in the main win- Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | mm) SfM error in sparse point cloud (Eq.1Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper | Discussion Paper |

Figure 1 .Figure 5 .Figure 8 .
Figure 1.SF3M v1.0 with the main window on the right, image window on the upper-left and display window on the bottom-left.Among other SF3M features, the figure shows: (1) highlighting of image listbox with detected ground control points GCPs (yellow) and with observations (blue) in the observations window; (2) text in green colour for performed operations;(3) subproject listbox for subproject management; (4) GCPs table with mean georef and control errors.A complete manual of SF3M operation can be found in http://sf3mapp.csic.esdomain.

Table 1 .
SF3M v1.0 features showing the aim of each stage, and software used.

Table 2 .
Field and processing time requirements for the entire gully network.Main results and type of operation are included.

Table 3 .
Comparison between the survey intensity (number of pictures per meter of gully) and time requirements (image collection time per meter of gully) for different gully erosion studies using SfM photo-reconstruction including the present study.