{"__v":202,"_id":"54e62e81a6972f0d002556d6","category":{"__v":16,"_id":"54e49219e4365521006fd9ee","pages":["54e4934e610dfd0d00b2a863","54e49857610dfd0d00b2a869","54e4d1b418155617007884de","54e4eaff1815561700788522","54e50932d71c112d005440e6","54e50982d71c112d005440e8","54e612b3cf61ff0d00cf95b6","54e62e81a6972f0d002556d6","54ec82cb5033a62b005b257a","54edebbe3749bf0d00c764f8","54ededd28dafa7250027e64d","54ee3be3ae79c92d000f41b4","54ee4c11df57670d00871c8e","54f9ba2167e8f20d0096da6f","54fdf1e47ed77417002c5384","55106b1a23eca20d0038d0dc"],"project":"54d0fd1d095c470d00d1646d","version":"54d0fd1e095c470d00d16470","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-02-18T13:22:33.459Z","from_sync":false,"order":0,"slug":"guides","title":"Guides"},"project":"54d0fd1d095c470d00d1646d","user":"54db646008ea010d00ab1fe1","version":{"__v":20,"_id":"54d0fd1e095c470d00d16470","project":"54d0fd1d095c470d00d1646d","createdAt":"2015-02-03T16:53:50.090Z","releaseDate":"2015-02-03T16:53:50.090Z","categories":["54d0fd1e095c470d00d16471","54d8b5e68934140d00496544","54db6361c6a4d40d0034b8f7","54db638208ea010d00ab1fdd","54db639008ea010d00ab1fde","54db6547c6a4d40d0034b8fd","54db83482092743700ea6ee1","54db84afc6a4d40d0034b93c","54db8805c6a4d40d0034b93f","54db8de9c6a4d40d0034b961","54db931ac6a4d40d0034b96d","54e49219e4365521006fd9ee","54e74fcc652d7c1900cbe737","54e74ffd3c1e111700d05762","54e77e0a523b1b2f00e6f313","54e784affdabe62500fcddcf","54e784fa523b1b2f00e6f319","54e785de8ae8911900cd42c5","54fdad6e660db63700c23b82","54fdff31f7b1202100a25e06"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"1.0.0","version":"1.0"},"updates":[],"next":{"pages":[],"description":""},"createdAt":"2015-02-19T18:42:09.296Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":5,"body":"Data import is a very basic and important step when dealing with databases. It allows your data to be inserted into **goGeo** platform, making it possible to manage data, create maps and all other cool stuff.\n\nThe data import procedure in the **goGeo** platform is performed following these steps:\n\n1. The user informs data file *URL* for importing;\n2. **goGeo** performs the download of this file;\n3. If necessary, the file is decompressed;\n4. So, insert the data on platform.\n\nIt is worth noting that documents may also be inserted individually through the use of a JSON structure, although for large amounts of data, it is more efficient to use a file with all your data.\n\nEach data registry inside **goGeo** platform is composed by relational and spatial data. The following tables describe the supported data type for both relational and spatial portion of data. \n\nThe relational data types supported by **goGeo** are:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"relational type\",\n    \"h-1\": \"Description\",\n    \"0-0\": \"int\",\n    \"1-0\": \"long\",\n    \"2-0\": \"float\",\n    \"3-0\": \"string\",\n    \"4-0\": \"date\",\n    \"4-1\": \"A string representing a date. All accepted data formats can be seen [here](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html).\",\n    \"3-1\": \"A character string.\",\n    \"2-1\": \"A floating point number.\",\n    \"1-1\": \"A long integer number.\",\n    \"0-1\": \"An integer number.\"\n  },\n  \"cols\": 2,\n  \"rows\": 5\n}\n[/block]\n\n[block:html]\n{\n  \"html\": \"<div class='footer_table'>\\nTable 1 - Supported relational data type.\\n</div>\\n\\n<style>\\n  .footer_table {\\n  \\ttext-align: center;\\n    max-width: 400px;\\n    margin: 10px auto 0px;\\n    color: #888;\\n    font-size: 0.9em;\\n    margin-top: -21px;\\n  }\\n</style>\"\n}\n[/block]\nThe following geometry types are supported by **goGeo**:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Geometry type\",\n    \"h-1\": \"Description\",\n    \"0-1\": \"Point representation in: *i)* WKT format; *ii)* in two float fields with the point coordinates (latlon); and *iii)* within a string field containing the point coordinates separated by a comma (location). The last two refer to import with CSV file.\",\n    \"1-1\": \"Line representation in WKT format.\",\n    \"2-1\": \"Polygon representation in WKT format.\",\n    \"3-1\": \"A geometry formed by one or more point geometries in WKT format.\",\n    \"4-1\": \"A geometry formed by one or more linestring geometries in WKT format.\",\n    \"5-1\": \"A geometry formed by one or more polygon geometries in WKT format.\",\n    \"0-0\": \"Point\",\n    \"1-0\": \"LineString\",\n    \"2-0\": \"Polygon\",\n    \"3-0\": \"MultiPoint\",\n    \"4-0\": \"MultiLineString\",\n    \"5-0\": \"MultiPolygon\"\n  },\n  \"cols\": 2,\n  \"rows\": 6\n}\n[/block]\n\n[block:html]\n{\n  \"html\": \"<div class='footer_table'>\\nTable 2 - Supported spatial data type.\\n</div>\\n\\n<style>\\n  .footer_table {\\n  \\ttext-align: center;\\n    max-width: 400px;\\n    margin: 10px auto 0px;\\n    color: #888;\\n    font-size: 0.9em;\\n    margin-top: -21px;\\n  }\\n</style>\"\n}\n[/block]\n## **File Import**\n\nWhen inserting many documents at once, the best option is making a file import. In this case, data is sent to **goGeo** through a publicly available *URL* pointing to where your file is (on a web-server or any cloud storage service).\n\nThe *URL* must be a direct link to the file. Below is described a command that you can use to test your *URL*.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"\\ncurl -L -i https://www.dropbox.com/s/rw0q82cu8ll2qcu/basic.csv?dl=1\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\nThe option *-i* of *curl* command denote that beyond getting the file, it should also show the request headers.\n\nIn the previous example we use a Dropbox shared file. Trying to access a Dropbox share *URL link* may lead to opening a preview page of the file instead of attempt to download it.\n\nTo cause the browser to download a file or folder from Dropbox rather than display it, you can use *dl=1* as a query parameter in your *URL*. Note that the original share *URL link* may contain query string parameters already (e.g. dl=0), so app developers should make sure to properly parse the URL and add or modify parameters as needed (i.e. set dl query parameter to 1!). Learn more [here](https://www.dropbox.com/en/help/201).\n\n**goGeo** platform accepts two types of files on data importing, [CSV](<http://pt.wikipedia.org/wiki/Comma-separated_values>) and [shapefiles](<http://en.wikipedia.org/wiki/Shapefile>).  See [Importing data](/v1.0/docs/importing-data#inserting-a-batch-of-documents-by-api)  tutorials to see how.\n\nA dataset in shapefile format is, in fact, composed by at least 3 files. All of them with the same name but distinct extension (**.dbf**, **.shx** and **.shp**). Thus, it is required compress these files together to perform the data import into **goGeo** platform.\n\nThe compressed formats accepted by the platform are: **.zip**, **.rar**, **.tar** and **.tar.bz2**.\n\nShapefile import is a simple task, since they contain all necessary information about the attributes and geometries and follow a well defined structure. It can be done inside Console or directly by API endpoints.\n\n## CSV format\n\nData import using a *CSV* file are in parts the same of importing a shapefile. It is required to inform the location (coordinates, for example). The file may also be compressed (using one of supported formats) to decrease the size of them. \n\nIn the other hand, there is no a unique standard format for *CSV* files. For this reason, when importing a *CSV* file a JSON metadata object is desirable in order to specify that format. If you don't include metadata, the platform will try to create a metadata for your file, but remember this process is not as accurate as yourself inform data types.\n\nThe following section gives more details about how that is done.\n\n## CSV Metadata\n\nThe format of JSON metadata will inform **goGeo** about the internal structure of the data and enable to work on it in the right way. Without that, the **goGeo** will read some lines of your file and try determine what type of each column. However the problems with parsing rows and columns may happen and even bad association of each file attributes to its correct type. Below is shown a *CSV* metadata sample.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n     \\\"field_delimiter\\\" : \\\";\\\",\\n     \\\"row_delimiter\\\" : \\\"\\\\n\\\",\\n     \\\"header\\\" : true,\\n     \\\"geom\\\" : {\\n          \\\"type\\\": \\\"latlon\\\",\\n          \\\"fields\\\" : [\\\"latitude\\\", \\\"longitude\\\"]\\n     },\\n     \\\"mapping\\\": [\\n          [\\\"field1\\\", \\\"string\\\"],\\n          [\\\"latitude\\\", \\\"float\\\"],\\n       \\t\\t[\\\"longitude\\\", \\\"float\\\"],\\n          [\\\"field2\\\", \\\"long\\\"]\\n     ]\\n}\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\nThis metadata sample contains information about a *CSV* file structure. As showed in metadata, the file uses semi-colon to separate columns and *'\\n'* as row delimiter. The fields are *field1*, *latitude*, *longitude* and *field2*. In addition, there is a header line (the very first line). \n\nThe geometries into the registers of this file are points. They are described by the parameters latitude and longitude located in the *latitude* and *longitude* fields. Further, they are indicated by the type of geom property of metadata object. \n\nThe first line of the file is a header line (indicated by the header parameter). Note that the file header is not used by **goGeo**, since the metadata JSON already contains all relevant information about the file data. Despite of this, if your data has a header line, it is necessary to set header equal to **true** in the metadata JSON.\n\nThe table below describe each property of the JSON metadata object:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Param\",\n    \"h-1\": \"Required?\",\n    \"h-2\": \"Default\",\n    \"0-0\": \"field_delimiter\",\n    \"0-1\": \"Yes\",\n    \"h-3\": \"Description\",\n    \"0-3\": \"The character used to separate the fields\",\n    \"0-2\": \"';'\",\n    \"1-0\": \"row_delimiter\",\n    \"1-3\": \"The character used to delimit lines. If not defined, the row delimiter is inferred by **goGeo**.\",\n    \"1-1\": \"No\",\n    \"1-2\": \"'\\\\n'\",\n    \"2-0\": \"header\",\n    \"2-1\": \"Yes\",\n    \"2-3\": \"Does your file has a header line at the beggining? Permitted values are **true** and **false**.\",\n    \"3-0\": \"geom\",\n    \"3-1\": \"Yes\",\n    \"3-3\": \"JSON object that describes the geometry field.\\ntype is the type of geometry your file contains.\\nfields is a list formed by the fields with the geometry information.\",\n    \"4-0\": \"date_format\",\n    \"4-1\": \"No\",\n    \"4-3\": \"JSON list that describes the format for date columns.\\nEach list element is another list with two elements.\\nthe first element is the date field name. The second one is the Date Format string applied to this field.\",\n    \"5-3\": \"JSON list that describes the file fields and their types.\\nEach list element is another list with two elements.\\nthe first element is the field name. The second one is the field type.\",\n    \"5-0\": \"mapping\",\n    \"5-1\": \"Yes\",\n    \"4-2\": \"If not informed, the engine attempts to identify the date format through a list of common formats.\"\n  },\n  \"cols\": 4,\n  \"rows\": 6\n}\n[/block]\n\n[block:html]\n{\n  \"html\": \"<div class='footer_table'>\\nTable 3 - Properties of JSON metadata object.\\n</div>\\n\\n<style>\\n  .footer_table {\\n  \\ttext-align: center;\\n    max-width: 400px;\\n    margin: 10px auto 0px;\\n    color: #888;\\n    font-size: 0.9em;\\n    margin-top: -21px;\\n  }\\n</style>\"\n}\n[/block]\nThe *geom* property of the metadata object is an important information. It tells the platform the type of the geometry and the field(s) where that geometry lies in. \n\nThe attribute *fields* is an array and the attribute *type* has there (3) possible values: *latlon*, *location* and *wkt*. Each of them are described below:\n\n*1.* latlon:\n\nThe *type* attribute is equal to **latlon**: indicates that the geometry type is \"Point\" and must exist two (2) float fields into \"mapping\" metadata property containing the latitude and longitude of the point. These fields are pointed by the attribute *fields*. Below, a metadata JSON object demonstrating a *latlon* geom sample:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n     \\\"field_delimiter\\\" : \\\";\\\",\\n     \\\"row_delimiter\\\" : \\\"\\\\n\\\",\\n     \\\"header\\\" : true,\\n     \\\"geom\\\" : {\\n          \\\"type\\\": \\\"latlon\\\",\\n          \\\"fields\\\" : [\\\"latitude\\\", \\\"longitude\\\"]\\n     },\\n     \\\"mapping\\\": [\\n          [\\\"field1\\\", \\\"string\\\"],\\n          [\\\"latitude\\\", \\\"float\\\"],\\n       \\t\\t[\\\"longitude\\\", \\\"float\\\"],\\n          [\\\"field2\\\", \\\"long\\\"]\\n     ]\\n}\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\n*2.* location:\n\nThe *type* attribute is equal to **location**: indicates that geometry type is \"Point\" and must exist exactly one string field into \"mapping\" metadata property containing the point coordinates. This field are indicated by the attribute *fields*. Below, a metadata JSON object demonstrating a *location* geom sample:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n     \\\"field_delimiter\\\" : \\\";\\\",\\n     \\\"row_delimiter\\\" : \\\"\\\\n\\\",\\n     \\\"header\\\" : true,\\n     \\\"geom\\\" : {\\n          \\\"type\\\": \\\"location\\\",\\n          \\\"fields\\\" : [\\\"coodinates\\\"]\\n     },\\n     \\\"mapping\\\": [\\n          [\\\"field1\\\", \\\"string\\\"],\\n          [\\\"coodinates\\\", \\\"String\\\"],\\n          [\\\"field2\\\", \\\"long\\\"]\\n     ]\\n}\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\n*3.* wkt types:\n\nIn the case of *WKT* type, there is a special feature for the *type* attribute. It will indicate two things at the same time: first it indicates that the geom attribute is of *WKT* kind, second it indicates the registry geometry type. This is done through the name assigned to the attribute type. The name is composed of the prefix \"wkt\" followed by a valid geometry type (see Table 2) separated by a underline character (\"_\"). The following is a valid value for the attribute *type*: \n\n*\"wkt_point\"*\n\nIn this case the type of registry geometry is the same informed after the underline (\"_\"). In the example above the geometry type is \"Point\".\n\nIf *type* attribute is equal to **wkt_polygon**: indicates that geometry type is \"Polygon\" and must exist one string field into \"mapping\" metadata property containing the wkt geometry specification. The field with the wkt geometry are indicated by the attribute *fields*. It is worth noting the type value for the field into \"mapping\" property must be the same value of the geom *type* attribute, as shown in example below.\n\nThe following sample shows a metadata JSON object for a *wkt_point* geom:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n     \\\"field_delimiter\\\" : \\\";\\\",\\n     \\\"row_delimiter\\\" : \\\"\\\\n\\\",\\n     \\\"header\\\" : true,\\n     \\\"geom\\\" : {\\n          \\\"type\\\": \\\"wkt_point\\\",\\n          \\\"fields\\\" : [\\\"the_geom\\\"]\\n     },\\n     \\\"mapping\\\": [\\n          [\\\"field1\\\", \\\"string\\\"],\\n          [\\\"the_geom\\\", \\\"wkt_point\\\"],\\n          [\\\"field2\\\", \\\"long\\\"]\\n     ]\\n}\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\nThe *date_format* property of the metadata file provides to the user the possibility to inform customized date formats or define different formats for different date fields. This property is not required, if user do not inform it, the engine will try to identify the correct format.\n\nIt is possible to define different format to different fields as shown on sample below. Other possibility is to inform date format for only part of the fields, the format to the fields not present into *date_format* property will be defined by the engine. \n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"\\\"date_format\\\" : [\\n  [\\\"date1\\\", \\\"yyyy-MM-dd'T'HH:mm:ss.S\\\"],\\n  [\\\"date2\\\", \\\"yyyy-MM-dd HH:mm:ss.S\\\"],\\n  [\\\"date3\\\", \\\"yyyy-MM-dd'T'HH:mm:ss\\\"],\\n  [\\\"date4\\\", \\\"yyyy-MM-dd HH:mm:ss\\\"],\\n  [\\\"date5\\\", \\\"dd/MM/yyyy\\\"]\\n],\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\nIf you want to assign the same date format for more than one field you need to include one item to each date field into the *date_format* list with the name of the field and date format string, even if the data formats are the same.\n\nTo any of the cases described here, if it is not possible to identify the date format, an exception will be thrown.\n\nThe *mapping* property of the metadata is a mapping of the fields of the csv file (as shown below), ie. it has an item for each column in the .csv file to be imported. It is important that the fields are informed in mapping in the same order and with equivalent types of the columns in the .csv file. \n\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"\\\"mapping\\\": [\\n  [\\\"field1\\\", \\\"string\\\"],\\n  [\\\"latitude\\\", \\\"float\\\"],\\n  [\\\"longitude\\\", \\\"float\\\"],\\n  [\\\"field2\\\", \\\"long\\\"]\\n],\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\n#**Document Import**\nThe easiest way to insert a single document into **goGeo** is using the [Document](https://dash.readme.io/project/gogeo/v1.0/docs/update-a-document)  API, through a JSON object representing the data you want to insert, like in the following example.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"json\\\" : [{\\n    \\\"geom\\\": {\\n      \\\"type\\\": \\\"Point\\\",\\n      \\\"coordinates\\\": [-118.141711, 34.181178]\\n    },\\n    \\\"properties\\\": {\\n      \\\"name\\\": \\\"Roberto\\\",\\n      \\\"age\\\": 38\\n    }\\n  }]\\n}\",\n      \"language\": \"json\"\n    }\n  ]\n}\n[/block]\nThe JSON object should have *geom* and *properties* as its attributes. *properties* is a JSON object representing the relational attributes. \n\n*geom* represent the document's geolocation, and The *geom* attribute is a GeoJSON object where the type member's value is one of the following strings: \"Point\", \"MultiPoint\", \"LineString\", \"MultiLineString\", \"Polygon\", \"MultiPolygon\", or \"GeometryCollection\" (**goGeo** support all of them, except the last one). \n\nA GeoJSON geometry object of any type other than \"GeometryCollection\" must have a member with the name \"coordinates\". The value of the coordinates member is always an array. The structure for the elements in this array is determined by the type of geometry. Learn more [here](http://geojson.org/geojson-spec.html#geometry-objects).\n\nThe JSON with key pair *geom* and *properties* form a document. You can pass many documents to insert at the same time. Its important that all documents has the same properties types and geom type.\n\nA full request for inserting a single document can be made like this:\n[block:callout]\n{\n  \"type\": \"danger\",\n  \"title\": \"NOTE: You need to create an account to get your own API-KEY.\"\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"curl --user API-KEY:\\\"\\\" -H \\\"Content-Type: application/json\\\" -XPUT \\\\\\n  \\\"https://api.gogeo.io/1.0/databases/my_database/collections/my_collection/documents/{document_id}\\\" -d '\\n{\\n  \\\"json\\\": [{\\n    \\\"geom\\\": {\\n      \\\"type\\\": \\\"Point\\\",\\n      \\\"coordinates\\\": [-118.141711, 34.181178]\\n    },\\n    \\\"properties\\\": {\\n      \\\"name\\\": \\\"Roberto\\\",\\n      \\\"age\\\": 38\\n    }\\n  }]\\n}'\",\n      \"language\": \"shell\"\n    }\n  ]\n}\n[/block]\n---\n[block:html]\n{\n  \"html\": \"<div class='div-middle'> \\n  <a href='#'>\\n    Top page &spades; </div>\\n  </a>\\n</div>\\n\\n<div class='div-back'> \\n  <a href='/v1.0/docs/data-model'>\\n    &laquo; Back \\n  </a>\\n</div>\\n\\n<div class='div-forward'> \\n  <a href='/v1.0/docs/styles-1'>\\n    Next &raquo; </div>\\n  </a>\\n</div>\\n\\n<style>\\n\\t.div-back {\\n  \\tpadding-left: 15px;\\n\\t\\tmargin-top: -20px;\\n  }\\n  \\n  .div-middle {\\n  \\ttext-align: center;\\n\\t\\tmargin-top: -15px;\\n  }\\n  \\n  .div-forward {\\n  \\tfloat: right;\\n    padding-right: 15px;\\n\\t\\tmargin-top: -20px;\\n  }\\n</style>\"\n}\n[/block]","excerpt":"","slug":"data-insertion","type":"basic","title":"Data Import"}
Data import is a very basic and important step when dealing with databases. It allows your data to be inserted into **goGeo** platform, making it possible to manage data, create maps and all other cool stuff. The data import procedure in the **goGeo** platform is performed following these steps: 1. The user informs data file *URL* for importing; 2. **goGeo** performs the download of this file; 3. If necessary, the file is decompressed; 4. So, insert the data on platform. It is worth noting that documents may also be inserted individually through the use of a JSON structure, although for large amounts of data, it is more efficient to use a file with all your data. Each data registry inside **goGeo** platform is composed by relational and spatial data. The following tables describe the supported data type for both relational and spatial portion of data. The relational data types supported by **goGeo** are: [block:parameters] { "data": { "h-0": "relational type", "h-1": "Description", "0-0": "int", "1-0": "long", "2-0": "float", "3-0": "string", "4-0": "date", "4-1": "A string representing a date. All accepted data formats can be seen [here](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html).", "3-1": "A character string.", "2-1": "A floating point number.", "1-1": "A long integer number.", "0-1": "An integer number." }, "cols": 2, "rows": 5 } [/block] [block:html] { "html": "<div class='footer_table'>\nTable 1 - Supported relational data type.\n</div>\n\n<style>\n .footer_table {\n \ttext-align: center;\n max-width: 400px;\n margin: 10px auto 0px;\n color: #888;\n font-size: 0.9em;\n margin-top: -21px;\n }\n</style>" } [/block] The following geometry types are supported by **goGeo**: [block:parameters] { "data": { "h-0": "Geometry type", "h-1": "Description", "0-1": "Point representation in: *i)* WKT format; *ii)* in two float fields with the point coordinates (latlon); and *iii)* within a string field containing the point coordinates separated by a comma (location). The last two refer to import with CSV file.", "1-1": "Line representation in WKT format.", "2-1": "Polygon representation in WKT format.", "3-1": "A geometry formed by one or more point geometries in WKT format.", "4-1": "A geometry formed by one or more linestring geometries in WKT format.", "5-1": "A geometry formed by one or more polygon geometries in WKT format.", "0-0": "Point", "1-0": "LineString", "2-0": "Polygon", "3-0": "MultiPoint", "4-0": "MultiLineString", "5-0": "MultiPolygon" }, "cols": 2, "rows": 6 } [/block] [block:html] { "html": "<div class='footer_table'>\nTable 2 - Supported spatial data type.\n</div>\n\n<style>\n .footer_table {\n \ttext-align: center;\n max-width: 400px;\n margin: 10px auto 0px;\n color: #888;\n font-size: 0.9em;\n margin-top: -21px;\n }\n</style>" } [/block] ## **File Import** When inserting many documents at once, the best option is making a file import. In this case, data is sent to **goGeo** through a publicly available *URL* pointing to where your file is (on a web-server or any cloud storage service). The *URL* must be a direct link to the file. Below is described a command that you can use to test your *URL*. [block:code] { "codes": [ { "code": "\ncurl -L -i https://www.dropbox.com/s/rw0q82cu8ll2qcu/basic.csv?dl=1", "language": "shell" } ] } [/block] The option *-i* of *curl* command denote that beyond getting the file, it should also show the request headers. In the previous example we use a Dropbox shared file. Trying to access a Dropbox share *URL link* may lead to opening a preview page of the file instead of attempt to download it. To cause the browser to download a file or folder from Dropbox rather than display it, you can use *dl=1* as a query parameter in your *URL*. Note that the original share *URL link* may contain query string parameters already (e.g. dl=0), so app developers should make sure to properly parse the URL and add or modify parameters as needed (i.e. set dl query parameter to 1!). Learn more [here](https://www.dropbox.com/en/help/201). **goGeo** platform accepts two types of files on data importing, [CSV](<http://pt.wikipedia.org/wiki/Comma-separated_values>) and [shapefiles](<http://en.wikipedia.org/wiki/Shapefile>). See [Importing data](/v1.0/docs/importing-data#inserting-a-batch-of-documents-by-api) tutorials to see how. A dataset in shapefile format is, in fact, composed by at least 3 files. All of them with the same name but distinct extension (**.dbf**, **.shx** and **.shp**). Thus, it is required compress these files together to perform the data import into **goGeo** platform. The compressed formats accepted by the platform are: **.zip**, **.rar**, **.tar** and **.tar.bz2**. Shapefile import is a simple task, since they contain all necessary information about the attributes and geometries and follow a well defined structure. It can be done inside Console or directly by API endpoints. ## CSV format Data import using a *CSV* file are in parts the same of importing a shapefile. It is required to inform the location (coordinates, for example). The file may also be compressed (using one of supported formats) to decrease the size of them. In the other hand, there is no a unique standard format for *CSV* files. For this reason, when importing a *CSV* file a JSON metadata object is desirable in order to specify that format. If you don't include metadata, the platform will try to create a metadata for your file, but remember this process is not as accurate as yourself inform data types. The following section gives more details about how that is done. ## CSV Metadata The format of JSON metadata will inform **goGeo** about the internal structure of the data and enable to work on it in the right way. Without that, the **goGeo** will read some lines of your file and try determine what type of each column. However the problems with parsing rows and columns may happen and even bad association of each file attributes to its correct type. Below is shown a *CSV* metadata sample. [block:code] { "codes": [ { "code": "{\n \"field_delimiter\" : \";\",\n \"row_delimiter\" : \"\\n\",\n \"header\" : true,\n \"geom\" : {\n \"type\": \"latlon\",\n \"fields\" : [\"latitude\", \"longitude\"]\n },\n \"mapping\": [\n [\"field1\", \"string\"],\n [\"latitude\", \"float\"],\n \t\t[\"longitude\", \"float\"],\n [\"field2\", \"long\"]\n ]\n}", "language": "json" } ] } [/block] This metadata sample contains information about a *CSV* file structure. As showed in metadata, the file uses semi-colon to separate columns and *'\n'* as row delimiter. The fields are *field1*, *latitude*, *longitude* and *field2*. In addition, there is a header line (the very first line). The geometries into the registers of this file are points. They are described by the parameters latitude and longitude located in the *latitude* and *longitude* fields. Further, they are indicated by the type of geom property of metadata object. The first line of the file is a header line (indicated by the header parameter). Note that the file header is not used by **goGeo**, since the metadata JSON already contains all relevant information about the file data. Despite of this, if your data has a header line, it is necessary to set header equal to **true** in the metadata JSON. The table below describe each property of the JSON metadata object: [block:parameters] { "data": { "h-0": "Param", "h-1": "Required?", "h-2": "Default", "0-0": "field_delimiter", "0-1": "Yes", "h-3": "Description", "0-3": "The character used to separate the fields", "0-2": "';'", "1-0": "row_delimiter", "1-3": "The character used to delimit lines. If not defined, the row delimiter is inferred by **goGeo**.", "1-1": "No", "1-2": "'\\n'", "2-0": "header", "2-1": "Yes", "2-3": "Does your file has a header line at the beggining? Permitted values are **true** and **false**.", "3-0": "geom", "3-1": "Yes", "3-3": "JSON object that describes the geometry field.\ntype is the type of geometry your file contains.\nfields is a list formed by the fields with the geometry information.", "4-0": "date_format", "4-1": "No", "4-3": "JSON list that describes the format for date columns.\nEach list element is another list with two elements.\nthe first element is the date field name. The second one is the Date Format string applied to this field.", "5-3": "JSON list that describes the file fields and their types.\nEach list element is another list with two elements.\nthe first element is the field name. The second one is the field type.", "5-0": "mapping", "5-1": "Yes", "4-2": "If not informed, the engine attempts to identify the date format through a list of common formats." }, "cols": 4, "rows": 6 } [/block] [block:html] { "html": "<div class='footer_table'>\nTable 3 - Properties of JSON metadata object.\n</div>\n\n<style>\n .footer_table {\n \ttext-align: center;\n max-width: 400px;\n margin: 10px auto 0px;\n color: #888;\n font-size: 0.9em;\n margin-top: -21px;\n }\n</style>" } [/block] The *geom* property of the metadata object is an important information. It tells the platform the type of the geometry and the field(s) where that geometry lies in. The attribute *fields* is an array and the attribute *type* has there (3) possible values: *latlon*, *location* and *wkt*. Each of them are described below: *1.* latlon: The *type* attribute is equal to **latlon**: indicates that the geometry type is "Point" and must exist two (2) float fields into "mapping" metadata property containing the latitude and longitude of the point. These fields are pointed by the attribute *fields*. Below, a metadata JSON object demonstrating a *latlon* geom sample: [block:code] { "codes": [ { "code": "{\n \"field_delimiter\" : \";\",\n \"row_delimiter\" : \"\\n\",\n \"header\" : true,\n \"geom\" : {\n \"type\": \"latlon\",\n \"fields\" : [\"latitude\", \"longitude\"]\n },\n \"mapping\": [\n [\"field1\", \"string\"],\n [\"latitude\", \"float\"],\n \t\t[\"longitude\", \"float\"],\n [\"field2\", \"long\"]\n ]\n}", "language": "json" } ] } [/block] *2.* location: The *type* attribute is equal to **location**: indicates that geometry type is "Point" and must exist exactly one string field into "mapping" metadata property containing the point coordinates. This field are indicated by the attribute *fields*. Below, a metadata JSON object demonstrating a *location* geom sample: [block:code] { "codes": [ { "code": "{\n \"field_delimiter\" : \";\",\n \"row_delimiter\" : \"\\n\",\n \"header\" : true,\n \"geom\" : {\n \"type\": \"location\",\n \"fields\" : [\"coodinates\"]\n },\n \"mapping\": [\n [\"field1\", \"string\"],\n [\"coodinates\", \"String\"],\n [\"field2\", \"long\"]\n ]\n}", "language": "json" } ] } [/block] *3.* wkt types: In the case of *WKT* type, there is a special feature for the *type* attribute. It will indicate two things at the same time: first it indicates that the geom attribute is of *WKT* kind, second it indicates the registry geometry type. This is done through the name assigned to the attribute type. The name is composed of the prefix "wkt" followed by a valid geometry type (see Table 2) separated by a underline character ("_"). The following is a valid value for the attribute *type*: *"wkt_point"* In this case the type of registry geometry is the same informed after the underline ("_"). In the example above the geometry type is "Point". If *type* attribute is equal to **wkt_polygon**: indicates that geometry type is "Polygon" and must exist one string field into "mapping" metadata property containing the wkt geometry specification. The field with the wkt geometry are indicated by the attribute *fields*. It is worth noting the type value for the field into "mapping" property must be the same value of the geom *type* attribute, as shown in example below. The following sample shows a metadata JSON object for a *wkt_point* geom: [block:code] { "codes": [ { "code": "{\n \"field_delimiter\" : \";\",\n \"row_delimiter\" : \"\\n\",\n \"header\" : true,\n \"geom\" : {\n \"type\": \"wkt_point\",\n \"fields\" : [\"the_geom\"]\n },\n \"mapping\": [\n [\"field1\", \"string\"],\n [\"the_geom\", \"wkt_point\"],\n [\"field2\", \"long\"]\n ]\n}", "language": "json" } ] } [/block] The *date_format* property of the metadata file provides to the user the possibility to inform customized date formats or define different formats for different date fields. This property is not required, if user do not inform it, the engine will try to identify the correct format. It is possible to define different format to different fields as shown on sample below. Other possibility is to inform date format for only part of the fields, the format to the fields not present into *date_format* property will be defined by the engine. [block:code] { "codes": [ { "code": "\"date_format\" : [\n [\"date1\", \"yyyy-MM-dd'T'HH:mm:ss.S\"],\n [\"date2\", \"yyyy-MM-dd HH:mm:ss.S\"],\n [\"date3\", \"yyyy-MM-dd'T'HH:mm:ss\"],\n [\"date4\", \"yyyy-MM-dd HH:mm:ss\"],\n [\"date5\", \"dd/MM/yyyy\"]\n],", "language": "json" } ] } [/block] If you want to assign the same date format for more than one field you need to include one item to each date field into the *date_format* list with the name of the field and date format string, even if the data formats are the same. To any of the cases described here, if it is not possible to identify the date format, an exception will be thrown. The *mapping* property of the metadata is a mapping of the fields of the csv file (as shown below), ie. it has an item for each column in the .csv file to be imported. It is important that the fields are informed in mapping in the same order and with equivalent types of the columns in the .csv file. [block:code] { "codes": [ { "code": "\"mapping\": [\n [\"field1\", \"string\"],\n [\"latitude\", \"float\"],\n [\"longitude\", \"float\"],\n [\"field2\", \"long\"]\n],", "language": "json" } ] } [/block] #**Document Import** The easiest way to insert a single document into **goGeo** is using the [Document](https://dash.readme.io/project/gogeo/v1.0/docs/update-a-document) API, through a JSON object representing the data you want to insert, like in the following example. [block:code] { "codes": [ { "code": "{\n \"json\" : [{\n \"geom\": {\n \"type\": \"Point\",\n \"coordinates\": [-118.141711, 34.181178]\n },\n \"properties\": {\n \"name\": \"Roberto\",\n \"age\": 38\n }\n }]\n}", "language": "json" } ] } [/block] The JSON object should have *geom* and *properties* as its attributes. *properties* is a JSON object representing the relational attributes. *geom* represent the document's geolocation, and The *geom* attribute is a GeoJSON object where the type member's value is one of the following strings: "Point", "MultiPoint", "LineString", "MultiLineString", "Polygon", "MultiPolygon", or "GeometryCollection" (**goGeo** support all of them, except the last one). A GeoJSON geometry object of any type other than "GeometryCollection" must have a member with the name "coordinates". The value of the coordinates member is always an array. The structure for the elements in this array is determined by the type of geometry. Learn more [here](http://geojson.org/geojson-spec.html#geometry-objects). The JSON with key pair *geom* and *properties* form a document. You can pass many documents to insert at the same time. Its important that all documents has the same properties types and geom type. A full request for inserting a single document can be made like this: [block:callout] { "type": "danger", "title": "NOTE: You need to create an account to get your own API-KEY." } [/block] [block:code] { "codes": [ { "code": "curl --user API-KEY:\"\" -H \"Content-Type: application/json\" -XPUT \\\n \"https://api.gogeo.io/1.0/databases/my_database/collections/my_collection/documents/{document_id}\" -d '\n{\n \"json\": [{\n \"geom\": {\n \"type\": \"Point\",\n \"coordinates\": [-118.141711, 34.181178]\n },\n \"properties\": {\n \"name\": \"Roberto\",\n \"age\": 38\n }\n }]\n}'", "language": "shell" } ] } [/block] --- [block:html] { "html": "<div class='div-middle'> \n <a href='#'>\n Top page &spades; </div>\n </a>\n</div>\n\n<div class='div-back'> \n <a href='/v1.0/docs/data-model'>\n &laquo; Back \n </a>\n</div>\n\n<div class='div-forward'> \n <a href='/v1.0/docs/styles-1'>\n Next &raquo; </div>\n </a>\n</div>\n\n<style>\n\t.div-back {\n \tpadding-left: 15px;\n\t\tmargin-top: -20px;\n }\n \n .div-middle {\n \ttext-align: center;\n\t\tmargin-top: -15px;\n }\n \n .div-forward {\n \tfloat: right;\n padding-right: 15px;\n\t\tmargin-top: -20px;\n }\n</style>" } [/block]