Extraction of Colombia, Brazil and Peru online data#
Run the tutorial via free cloud platforms: 
This notebook provides you with an introduction on how to retrieve data from online data sources through web scraping, as well as from local files from Colombia, Brazil, Peru, and the Dominican Republic. This tutorial assumes you have an intermediate or advanced understanding of Python and data manipulation.
Setting up the environment#
To run this notebook, you need to have the following prerequisites:
Python 3.10+
Additionally, you need to install the socio4health and pandas package, which can be done using pip:
!pip install socio4health pandas -q
[notice] A new release of pip is available: 25.1.1 -> 25.2
[notice] To update, run: python.exe -m pip install --upgrade pip
Import Libraries#
To perform the data extraction, the socio4health library provides the Extractor class for data extraction, and the Harmonizer class for data harmonization of the retrieved date. We will also use pandas for data manipulation.
from socio4health import Extractor
from socio4health.enums.data_info_enum import BraColnamesEnum, BraColspecsEnum
Use case 1: Extracting data from Colombia#
To extract data from Colombia, we will use the Extractor class from the socio4health library. The Extractor class provides methods to retrieve data from various sources, including online databases and local files. In this example, we will extract the Large Integrated Household Survey - GEIH - 2022 (Gran Encuesta Integrada de Hogares - GEIH - 2022) dataset from the Colombian Nacional Administration of Statistics (DANE) website
The Extractor class requires the following parameters:
input_path: TheURLor local path to the data source.down_ext: A list of file extensions to download. This can include.CSV,.csv,.zip, etc.sep: The separator used in the data files (e.g.,;for semicolon-separated values).output_path: The local path where the extracted data will be saved.depth: The depth of the directory structure to traverse when downloading files. A depth of0means only the files in the specified directory will be downloaded.
col_online_extractor = Extractor(input_path="https://microdatos.dane.gov.co/index.php/catalog/771/get-microdata", down_ext=['.CSV','.csv','.zip'], sep=';', output_path="../data", depth=0)
After the instance is set up, we can call the s4h_extract method to download and extract the data. The method returns a list of pandas DataFrames containing the extracted data.
col_dfs = col_online_extractor.s4h_extract()
2026-10-08 09:59:11,249 - INFO - ----------------------
2026-10-08 09:59:11,249 - INFO - Starting data extraction...
2026-10-08 09:59:11,250 - INFO - Extracting data in online mode...
2026-10-08 09:59:11,250 - INFO - Scraping URL: https://microdatos.dane.gov.co/index.php/catalog/771/get-microdata with depth 0
2026-10-08 09:59:13,151 - INFO - Spider completed successfully for URL: https://microdatos.dane.gov.co/index.php/catalog/771/get-microdata
2026-10-08 09:59:13,161 - INFO - Downloading files to: ../data
Downloading files: 0%| | 0/12 [00:00<?, ?it/s]2026-10-08 09:59:15,324 - INFO - Successfully downloaded: GEIH_Enero_2022_Marco_2018.zip
Downloading files: 8%|████▊ | 1/12 [00:02<00:23, 2.16s/it]2026-10-08 09:59:17,501 - INFO - Successfully downloaded: GEIH_Febrero_2022_Marco_2018.zip
Downloading files: 17%|█████████▋ | 2/12 [00:04<00:21, 2.17s/it]2026-10-08 09:59:19,706 - INFO - Successfully downloaded: GEIH_Marzo_2022_Marco_2018.zip
Downloading files: 25%|██████████████▌ | 3/12 [00:06<00:19, 2.19s/it]2026-10-08 09:59:21,181 - INFO - Successfully downloaded: GEIH_Mayo_2022_Marco_2018.zip
Downloading files: 33%|███████████████████▎ | 4/12 [00:08<00:15, 1.91s/it]2026-10-08 09:59:23,172 - INFO - Successfully downloaded: GEIH_Junio_2022_Marco_2018.zip
Downloading files: 42%|████████████████████████▏ | 5/12 [00:10<00:13, 1.94s/it]2026-10-08 09:59:25,116 - INFO - Successfully downloaded: GEIH_Julio_2022_Marco_2018.zip
Downloading files: 50%|█████████████████████████████ | 6/12 [00:11<00:11, 1.94s/it]2026-10-08 09:59:27,134 - INFO - Successfully downloaded: GEIH_Agosto_2022_Marco_2018.zip
Downloading files: 58%|█████████████████████████████████▊ | 7/12 [00:13<00:09, 1.96s/it]2026-10-08 09:59:29,267 - INFO - Successfully downloaded: GEIH_Septiembre_Marco_2018.zip
Downloading files: 67%|██████████████████████████████████████▋ | 8/12 [00:16<00:08, 2.02s/it]2026-10-08 09:59:31,294 - INFO - Successfully downloaded: GEIH_Octubre_Marco_2018.zip
Downloading files: 75%|███████████████████████████████████████████▌ | 9/12 [00:18<00:06, 2.02s/it]2026-10-08 09:59:33,239 - INFO - Successfully downloaded: GEIH_Diciembre_2022_Marco_2018.zip
Downloading files: 83%|███████████████████████████████████████████████▌ | 10/12 [00:20<00:03, 2.00s/it]2026-10-08 09:59:35,427 - INFO - Successfully downloaded: GEIH_Abril_2022_Marco_2018_Act.zip
Downloading files: 92%|████████████████████████████████████████████████████▎ | 11/12 [00:22<00:02, 2.06s/it]2026-10-08 09:59:37,471 - INFO - Successfully downloaded: GEIH_Noviembre_2022_Marco_2018.act.zip
Downloading files: 100%|█████████████████████████████████████████████████████████| 12/12 [00:24<00:00, 2.03s/it]
2026-10-08 09:59:37,475 - INFO - Processing (depth 0): GEIH_Enero_2022_Marco_2018.zip
2026-10-08 10:03:08,328 - INFO - Extracted: c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Características generales, seguridad social en salud y educación.csv
2026-10-08 10:03:08,330 - INFO - Extracted: c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Datos del hogar y la vivienda.CSV
2026-10-08 10:03:08,335 - INFO - Extracted: c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Fuerza de trabajo.csv
2026-10-08 10:03:08,338 - INFO - Extracted: c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Migración.CSV
2026-10-08 10:03:08,340 - INFO - Extracted: c8bca186_GEIH_Enero_2022_Marco_2018_CSV_No ocupados.csv
2026-10-08 10:03:08,345 - INFO - Extracted: c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Ocupados.csv
2026-10-08 10:03:08,349 - INFO - Extracted: c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Otras formas de trabajo.csv
2026-10-08 10:03:08,352 - INFO - Extracted: c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Otros ingresos e impuestos.csv
2026-10-08 10:03:08,354 - INFO - Extracted: c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Tipo de investigación.csv
2026-10-08 10:03:15,622 - INFO - Processing (depth 0): GEIH_Febrero_2022_Marco_2018.zip
2026-10-08 10:04:04,321 - INFO - Extracted: fb8de51c_GEIH_Febrero_2022_Marco_2018_CSV_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:04:04,325 - INFO - Extracted: fb8de51c_GEIH_Febrero_2022_Marco_2018_CSV_Datos del hogar y la vivienda.CSV
2026-10-08 10:04:04,328 - INFO - Extracted: fb8de51c_GEIH_Febrero_2022_Marco_2018_CSV_Fuerza de trabajo.CSV
2026-10-08 10:04:04,332 - INFO - Extracted: fb8de51c_GEIH_Febrero_2022_Marco_2018_CSV_Migración.CSV
2026-10-08 10:04:04,334 - INFO - Extracted: fb8de51c_GEIH_Febrero_2022_Marco_2018_CSV_No ocupados.CSV
2026-10-08 10:04:04,339 - INFO - Extracted: fb8de51c_GEIH_Febrero_2022_Marco_2018_CSV_Ocupados.CSV
2026-10-08 10:04:04,344 - INFO - Extracted: fb8de51c_GEIH_Febrero_2022_Marco_2018_CSV_Otras formas de trabajo.CSV
2026-10-08 10:04:04,347 - INFO - Extracted: fb8de51c_GEIH_Febrero_2022_Marco_2018_CSV_Otros ingresos e impuestos.CSV
2026-10-08 10:04:04,350 - INFO - Extracted: fb8de51c_GEIH_Febrero_2022_Marco_2018_CSV_Tipo de investigación.CSV
2026-10-08 10:04:07,943 - INFO - Processing (depth 0): GEIH_Marzo_2022_Marco_2018.zip
2026-10-08 10:04:57,753 - INFO - Extracted: 0e57b75b_GEIH_Marzo_2022_Marco_2018_csv_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:04:57,755 - INFO - Extracted: 0e57b75b_GEIH_Marzo_2022_Marco_2018_csv_Datos del hogar y la vivienda.CSV
2026-10-08 10:04:57,758 - INFO - Extracted: 0e57b75b_GEIH_Marzo_2022_Marco_2018_csv_Fuerza de trabajo.CSV
2026-10-08 10:04:57,762 - INFO - Extracted: 0e57b75b_GEIH_Marzo_2022_Marco_2018_csv_Migración.CSV
2026-10-08 10:04:57,764 - INFO - Extracted: 0e57b75b_GEIH_Marzo_2022_Marco_2018_csv_No ocupados.CSV
2026-10-08 10:04:57,769 - INFO - Extracted: 0e57b75b_GEIH_Marzo_2022_Marco_2018_csv_Ocupados.CSV
2026-10-08 10:04:57,774 - INFO - Extracted: 0e57b75b_GEIH_Marzo_2022_Marco_2018_csv_Otras formas de trabajo.CSV
2026-10-08 10:04:57,777 - INFO - Extracted: 0e57b75b_GEIH_Marzo_2022_Marco_2018_csv_Otros ingresos e impuestos.CSV
2026-10-08 10:04:57,779 - INFO - Extracted: 0e57b75b_GEIH_Marzo_2022_Marco_2018_csv_Tipo de investigación.CSV
2026-10-08 10:04:58,993 - INFO - Processing (depth 0): GEIH_Mayo_2022_Marco_2018.zip
2026-10-08 10:04:59,369 - INFO - Processing (depth 1): CSV.zip
2026-10-08 10:04:59,778 - INFO - Extracted: 346cc6b8_csv_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:04:59,781 - INFO - Extracted: 346cc6b8_csv_Datos del hogar y la vivienda.CSV
2026-10-08 10:04:59,784 - INFO - Extracted: 346cc6b8_csv_Fuerza de trabajo.CSV
2026-10-08 10:04:59,788 - INFO - Extracted: 346cc6b8_csv_Migración.CSV
2026-10-08 10:04:59,791 - INFO - Extracted: 346cc6b8_csv_No ocupados.CSV
2026-10-08 10:04:59,795 - INFO - Extracted: 346cc6b8_csv_Ocupados.CSV
2026-10-08 10:04:59,800 - INFO - Extracted: 346cc6b8_csv_Otras formas de trabajo.CSV
2026-10-08 10:04:59,804 - INFO - Extracted: 346cc6b8_csv_Otros ingresos e impuestos.CSV
2026-10-08 10:04:59,807 - INFO - Extracted: 346cc6b8_csv_Tipo de investigación.CSV
2026-10-08 10:04:59,808 - INFO - Processing (depth 1): DTA.zip
2026-10-08 10:05:10,976 - WARNING - No matches in DTA.zip. Contents:
2026-10-08 10:05:10,976 - WARNING - No files found matching the specified extensions.
2026-10-08 10:05:10,977 - INFO - Processing (depth 1): SAV.zip
2026-10-08 10:05:17,028 - WARNING - No matches in SAV.zip. Contents:
2026-10-08 10:05:17,029 - WARNING - No files found matching the specified extensions.
2026-10-08 10:05:17,038 - INFO - Processing (depth 0): GEIH_Junio_2022_Marco_2018.zip
2026-10-08 10:06:08,574 - INFO - Extracted: 37710ed1_GEIH_Junio_2022_Marco_2018_CSV(1)_CSV_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:06:08,578 - INFO - Extracted: 37710ed1_GEIH_Junio_2022_Marco_2018_CSV(1)_CSV_Datos del hogar y la vivienda.CSV
2026-10-08 10:06:08,582 - INFO - Extracted: 37710ed1_GEIH_Junio_2022_Marco_2018_CSV(1)_CSV_Fuerza de trabajo.CSV
2026-10-08 10:06:08,586 - INFO - Extracted: 37710ed1_GEIH_Junio_2022_Marco_2018_CSV(1)_CSV_Migración.CSV
2026-10-08 10:06:08,589 - INFO - Extracted: 37710ed1_GEIH_Junio_2022_Marco_2018_CSV(1)_CSV_No ocupados.CSV
2026-10-08 10:06:08,595 - INFO - Extracted: 37710ed1_GEIH_Junio_2022_Marco_2018_CSV(1)_CSV_Ocupados.CSV
2026-10-08 10:06:08,600 - INFO - Extracted: 37710ed1_GEIH_Junio_2022_Marco_2018_CSV(1)_CSV_Otras formas de trabajo.CSV
2026-10-08 10:06:08,604 - INFO - Extracted: 37710ed1_GEIH_Junio_2022_Marco_2018_CSV(1)_CSV_Otros ingresos e impuestos.CSV
2026-10-08 10:06:08,607 - INFO - Extracted: 37710ed1_GEIH_Junio_2022_Marco_2018_CSV(1)_CSV_Tipo de investigación.CSV
2026-10-08 10:06:13,118 - INFO - Processing (depth 0): GEIH_Julio_2022_Marco_2018.zip
2026-10-08 10:07:36,771 - INFO - Extracted: 7a9c1851_GEIH_Julio_2022_Marco_2018_GEIH_Julio_Marco_2018_CSV_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:07:36,777 - INFO - Extracted: 7a9c1851_GEIH_Julio_2022_Marco_2018_GEIH_Julio_Marco_2018_CSV_Datos del hogar y la vivienda.CSV
2026-10-08 10:07:36,781 - INFO - Extracted: 7a9c1851_GEIH_Julio_2022_Marco_2018_GEIH_Julio_Marco_2018_CSV_Fuerza de trabajo.CSV
2026-10-08 10:07:36,785 - INFO - Extracted: 7a9c1851_GEIH_Julio_2022_Marco_2018_GEIH_Julio_Marco_2018_CSV_Migración.CSV
2026-10-08 10:07:36,788 - INFO - Extracted: 7a9c1851_GEIH_Julio_2022_Marco_2018_GEIH_Julio_Marco_2018_CSV_No ocupados.CSV
2026-10-08 10:07:36,793 - INFO - Extracted: 7a9c1851_GEIH_Julio_2022_Marco_2018_GEIH_Julio_Marco_2018_CSV_Ocupados.CSV
2026-10-08 10:07:36,799 - INFO - Extracted: 7a9c1851_GEIH_Julio_2022_Marco_2018_GEIH_Julio_Marco_2018_CSV_Otras formas de trabajo.CSV
2026-10-08 10:07:36,802 - INFO - Extracted: 7a9c1851_GEIH_Julio_2022_Marco_2018_GEIH_Julio_Marco_2018_CSV_Otros ingresos e impuestos.CSV
2026-10-08 10:07:36,806 - INFO - Extracted: 7a9c1851_GEIH_Julio_2022_Marco_2018_GEIH_Julio_Marco_2018_CSV_Tipo de investigación.CSV
2026-10-08 10:07:37,971 - INFO - Processing (depth 0): GEIH_Agosto_2022_Marco_2018.zip
2026-10-08 10:08:26,211 - INFO - Extracted: c80488e3_GEIH_Agosto_2022_Marco_2018_CSV_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:08:26,213 - INFO - Extracted: c80488e3_GEIH_Agosto_2022_Marco_2018_CSV_Datos del hogar y la vivienda.CSV
2026-10-08 10:08:26,216 - INFO - Extracted: c80488e3_GEIH_Agosto_2022_Marco_2018_CSV_Fuerza de trabajo.CSV
2026-10-08 10:08:26,220 - INFO - Extracted: c80488e3_GEIH_Agosto_2022_Marco_2018_CSV_Migración.CSV
2026-10-08 10:08:26,222 - INFO - Extracted: c80488e3_GEIH_Agosto_2022_Marco_2018_CSV_No ocupados.CSV
2026-10-08 10:08:26,227 - INFO - Extracted: c80488e3_GEIH_Agosto_2022_Marco_2018_CSV_Ocupados.CSV
2026-10-08 10:08:26,231 - INFO - Extracted: c80488e3_GEIH_Agosto_2022_Marco_2018_CSV_Otras formas de trabajo.CSV
2026-10-08 10:08:26,235 - INFO - Extracted: c80488e3_GEIH_Agosto_2022_Marco_2018_CSV_Otros ingresos e impuestos.CSV
2026-10-08 10:08:26,237 - INFO - Extracted: c80488e3_GEIH_Agosto_2022_Marco_2018_CSV_Tipo de investigación.CSV
2026-10-08 10:08:27,044 - INFO - Processing (depth 0): GEIH_Septiembre_Marco_2018.zip
2026-10-08 10:08:27,105 - INFO - Processing (depth 1): CSV.zip
2026-10-08 10:08:27,266 - INFO - Extracted: f05b592a_CSV_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:08:27,269 - INFO - Extracted: f05b592a_CSV_Datos del hogar y la vivienda.CSV
2026-10-08 10:08:27,273 - INFO - Extracted: f05b592a_CSV_Fuerza de trabajo.CSV
2026-10-08 10:08:27,276 - INFO - Extracted: f05b592a_CSV_Migración.CSV
2026-10-08 10:08:27,279 - INFO - Extracted: f05b592a_CSV_No ocupados.CSV
2026-10-08 10:08:27,283 - INFO - Extracted: f05b592a_CSV_Ocupados.CSV
2026-10-08 10:08:27,287 - INFO - Extracted: f05b592a_CSV_Otras formas de trabajo.CSV
2026-10-08 10:08:27,290 - INFO - Extracted: f05b592a_CSV_Otros ingresos e impuestos.CSV
2026-10-08 10:08:27,293 - INFO - Extracted: f05b592a_CSV_Tipo de investigación.CSV
2026-10-08 10:08:27,294 - INFO - Processing (depth 1): DTA.zip
2026-10-08 10:08:48,811 - WARNING - No matches in DTA.zip. Contents:
2026-10-08 10:08:48,811 - WARNING - No files found matching the specified extensions.
2026-10-08 10:08:48,812 - INFO - Processing (depth 1): SAV.zip
2026-10-08 10:09:13,505 - WARNING - No matches in SAV.zip. Contents:
2026-10-08 10:09:13,505 - WARNING - No files found matching the specified extensions.
2026-10-08 10:09:13,513 - INFO - Processing (depth 0): GEIH_Octubre_Marco_2018.zip
2026-10-08 10:09:13,568 - INFO - Processing (depth 1): CSV 4.zip
2026-10-08 10:09:13,675 - INFO - Extracted: 1e59b2fb_CSV_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:09:13,677 - INFO - Extracted: 1e59b2fb_CSV_Datos del hogar y la vivienda.CSV
2026-10-08 10:09:13,680 - INFO - Extracted: 1e59b2fb_CSV_Fuerza de trabajo.CSV
2026-10-08 10:09:13,684 - INFO - Extracted: 1e59b2fb_CSV_Migración.CSV
2026-10-08 10:09:13,686 - INFO - Extracted: 1e59b2fb_CSV_No ocupados.CSV
2026-10-08 10:09:13,690 - INFO - Extracted: 1e59b2fb_CSV_Ocupados.CSV
2026-10-08 10:09:13,695 - INFO - Extracted: 1e59b2fb_CSV_Otras formas de trabajo.CSV
2026-10-08 10:09:13,698 - INFO - Extracted: 1e59b2fb_CSV_Otros ingresos e impuestos.CSV
2026-10-08 10:09:13,700 - INFO - Extracted: 1e59b2fb_CSV_Tipo de investigación.CSV
2026-10-08 10:09:13,701 - INFO - Processing (depth 1): DTA 4.zip
2026-10-08 10:09:35,679 - WARNING - No matches in DTA 4.zip. Contents:
2026-10-08 10:09:35,679 - WARNING - No files found matching the specified extensions.
2026-10-08 10:09:35,680 - INFO - Processing (depth 1): SAV 4.zip
2026-10-08 10:09:56,404 - WARNING - No matches in SAV 4.zip. Contents:
2026-10-08 10:09:56,404 - WARNING - No files found matching the specified extensions.
2026-10-08 10:09:56,414 - INFO - Processing (depth 0): GEIH_Diciembre_2022_Marco_2018.zip
2026-10-08 10:10:42,546 - INFO - Extracted: 5881e2ed_GEIH_Diciembre_2022_Marco_2018_CVS_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:10:42,549 - INFO - Extracted: 5881e2ed_GEIH_Diciembre_2022_Marco_2018_CVS_Datos del hogar y la vivienda.CSV
2026-10-08 10:10:42,552 - INFO - Extracted: 5881e2ed_GEIH_Diciembre_2022_Marco_2018_CVS_Fuerza de trabajo.CSV
2026-10-08 10:10:42,555 - INFO - Extracted: 5881e2ed_GEIH_Diciembre_2022_Marco_2018_CVS_Migración.CSV
2026-10-08 10:10:42,558 - INFO - Extracted: 5881e2ed_GEIH_Diciembre_2022_Marco_2018_CVS_No ocupados.CSV
2026-10-08 10:10:42,562 - INFO - Extracted: 5881e2ed_GEIH_Diciembre_2022_Marco_2018_CVS_Ocupados.CSV
2026-10-08 10:10:42,567 - INFO - Extracted: 5881e2ed_GEIH_Diciembre_2022_Marco_2018_CVS_Otras formas de trabajo.CSV
2026-10-08 10:10:42,570 - INFO - Extracted: 5881e2ed_GEIH_Diciembre_2022_Marco_2018_CVS_Otros ingresos e impuestos.CSV
2026-10-08 10:10:42,572 - INFO - Extracted: 5881e2ed_GEIH_Diciembre_2022_Marco_2018_CVS_Tipo de investigación.CSV
2026-10-08 10:10:43,812 - INFO - Processing (depth 0): GEIH_Abril_2022_Marco_2018_Act.zip
2026-10-08 10:11:36,783 - INFO - Extracted: 4079475d_GEIH_Abril_2022_Marco_2018_CSV_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:11:36,785 - INFO - Extracted: 4079475d_GEIH_Abril_2022_Marco_2018_CSV_Datos del hogar y la vivienda.CSV
2026-10-08 10:11:36,788 - INFO - Extracted: 4079475d_GEIH_Abril_2022_Marco_2018_CSV_Fuerza de trabajo.CSV
2026-10-08 10:11:36,792 - INFO - Extracted: 4079475d_GEIH_Abril_2022_Marco_2018_CSV_Migración.CSV
2026-10-08 10:11:36,795 - INFO - Extracted: 4079475d_GEIH_Abril_2022_Marco_2018_CSV_No ocupados.CSV
2026-10-08 10:11:36,800 - INFO - Extracted: 4079475d_GEIH_Abril_2022_Marco_2018_CSV_Ocupados.CSV
2026-10-08 10:11:36,804 - INFO - Extracted: 4079475d_GEIH_Abril_2022_Marco_2018_CSV_Otras formas de trabajo.CSV
2026-10-08 10:11:36,808 - INFO - Extracted: 4079475d_GEIH_Abril_2022_Marco_2018_CSV_Otros ingresos e impuestos.CSV
2026-10-08 10:11:36,810 - INFO - Extracted: 4079475d_GEIH_Abril_2022_Marco_2018_CSV_Tipo de investigación.CSV
2026-10-08 10:11:37,800 - INFO - Processing (depth 0): GEIH_Noviembre_2022_Marco_2018.act.zip
2026-10-08 10:12:16,031 - INFO - Extracted: 0d034efb_GEIH_Noviembre_2022_Marco_2018_CSV 5_CSV_Características generales, seguridad social en salud y educación.CSV
2026-10-08 10:12:16,034 - INFO - Extracted: 0d034efb_GEIH_Noviembre_2022_Marco_2018_CSV 5_CSV_Datos del hogar y la vivienda.CSV
2026-10-08 10:12:16,037 - INFO - Extracted: 0d034efb_GEIH_Noviembre_2022_Marco_2018_CSV 5_CSV_Fuerza de trabajo.CSV
2026-10-08 10:12:16,041 - INFO - Extracted: 0d034efb_GEIH_Noviembre_2022_Marco_2018_CSV 5_CSV_Migración.CSV
2026-10-08 10:12:16,045 - INFO - Extracted: 0d034efb_GEIH_Noviembre_2022_Marco_2018_CSV 5_CSV_No ocupados.CSV
2026-10-08 10:12:16,051 - INFO - Extracted: 0d034efb_GEIH_Noviembre_2022_Marco_2018_CSV 5_CSV_Ocupados.CSV
2026-10-08 10:12:16,057 - INFO - Extracted: 0d034efb_GEIH_Noviembre_2022_Marco_2018_CSV 5_CSV_Otras formas de trabajo.CSV
2026-10-08 10:12:16,060 - INFO - Extracted: 0d034efb_GEIH_Noviembre_2022_Marco_2018_CSV 5_CSV_Otros ingresos e impuestos.CSV
2026-10-08 10:12:16,063 - INFO - Extracted: 0d034efb_GEIH_Noviembre_2022_Marco_2018_CSV 5_CSV_Tipo de investigación.CSV
Processing files: 100%|████████████████████████████████████████████████████████| 108/108 [00:13<00:00, 7.75it/s]
2026-10-08 10:12:30,920 - INFO - Successfully processed 108/108 files
2026-10-08 10:12:30,921 - INFO - Extraction completed successfully.
col_dfs[0].head()
| PERIODO | DIRECTORIO | SECUENCIA_P | ORDEN | HOGAR | P3076S1 | P3076S1A1 | P3076S1A2 | P3076S2 | P3076S2A1 | ... | AREA | MES | DPTO | P3091 | P3092 | P3093 | FEX_C18 | PER | REGIS | filename | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 20220104 | 5000000 | 1 | 1 | 1 | 1 | 3 | 2 | 2 | <NA> | ... | <NA> | 1 | 08 | 2 | <NA> | 2 | 1432.4633227 | 2022 | 80 | c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Otras ... |
| 1 | 20220104 | 5000000 | 1 | 2 | 1 | 1 | 7 | 4 | 2 | <NA> | ... | <NA> | 1 | 08 | 2 | <NA> | 2 | 1432.4633227 | 2022 | 80 | c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Otras ... |
| 2 | 20220104 | 5000000 | 1 | 6 | 1 | 2 | <NA> | <NA> | 2 | <NA> | ... | <NA> | 1 | 08 | 2 | <NA> | 2 | 1432.4633227 | 2022 | 80 | c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Otras ... |
| 3 | 20220104 | 5000001 | 1 | 1 | 1 | 2 | <NA> | <NA> | 2 | <NA> | ... | <NA> | 1 | 08 | 2 | <NA> | 2 | 1088.7962663 | 2022 | 80 | c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Otras ... |
| 4 | 20220104 | 5000001 | 1 | 2 | 1 | 1 | 7 | 2 | 2 | <NA> | ... | <NA> | 1 | 08 | 2 | <NA> | 2 | 1088.7962663 | 2022 | 80 | c8bca186_GEIH_Enero_2022_Marco_2018_CSV_Otras ... |
5 rows × 113 columns
Use case 2: Extracting data from Brazil#
We are downloading the Brazilian data from the Brazilian Institute of Geography and Statistics (IBGE) website. The Extractor class is used to download the data. In this case, we are extracting the Brazilian National Continuous Household Sample Survey (PNADC) for the year 2024
⚠️
Important:
is_fwf parameter is set to True, which indicates that the data files are in fixed-width format. The colnames and colspecs parameters must be provided. In this example, they are set to the corresponding available enums for PNADC data, which define the column names and specifications for the dataset.
See more details in
socio4health.enums.data_info_enum documentation
.
bra_online_extractor = Extractor(input_path="https://ftp.ibge.gov.br/Trabalho_e_Rendimento/Pesquisa_Nacional_por_Amostra_de_Domicilios_continua/Trimestral/Microdados/2024/", down_ext=['.txt','.zip'], is_fwf=True, colnames=BraColnamesEnum.PNADC.value, colspecs=BraColspecsEnum.PNADC.value, output_path="../data", depth=0)
bra_dfs = bra_online_extractor.s4h_extract()
2026-10-08 10:18:47,691 - INFO - ----------------------
2026-10-08 10:18:47,692 - INFO - Starting data extraction...
2026-10-08 10:18:47,692 - INFO - Extracting data in online mode...
2026-10-08 10:18:47,692 - INFO - Scraping URL: https://ftp.ibge.gov.br/Trabalho_e_Rendimento/Pesquisa_Nacional_por_Amostra_de_Domicilios_continua/Trimestral/Microdados/2024/ with depth 0
2026-10-08 10:18:52,541 - INFO - Spider completed successfully for URL: https://ftp.ibge.gov.br/Trabalho_e_Rendimento/Pesquisa_Nacional_por_Amostra_de_Domicilios_continua/Trimestral/Microdados/2024/
2026-10-08 10:18:52,551 - INFO - Downloading files to: ../data
Downloading files: 0%| | 0/4 [00:00<?, ?it/s]2026-10-08 10:19:07,281 - INFO - Successfully downloaded: PNADC_012024_20250815.zip
Downloading files: 25%|██████████████▊ | 1/4 [00:14<00:44, 14.73s/it]2026-10-08 10:19:22,176 - INFO - Successfully downloaded: PNADC_022024_20260324.zip
Downloading files: 50%|█████████████████████████████▌ | 2/4 [00:29<00:29, 14.83s/it]2026-10-08 10:19:37,052 - INFO - Successfully downloaded: PNADC_032024_20250815.zip
Downloading files: 75%|████████████████████████████████████████████▎ | 3/4 [00:44<00:14, 14.85s/it]2026-10-08 10:19:50,619 - INFO - Successfully downloaded: PNADC_042024_20250815.zip
Downloading files: 100%|███████████████████████████████████████████████████████████| 4/4 [00:58<00:00, 14.52s/it]
2026-10-08 10:19:50,622 - INFO - Processing (depth 0): PNADC_012024_20250815.zip
2026-10-08 10:20:09,364 - INFO - Extracted: a7db871d_PNADC_012024.txt
2026-10-08 10:20:09,366 - INFO - Processing (depth 0): PNADC_022024_20260324.zip
2026-10-08 10:20:24,277 - INFO - Extracted: 1ef780fa_PNADC_022024.txt
2026-10-08 10:20:24,278 - INFO - Processing (depth 0): PNADC_032024_20250815.zip
2026-10-08 10:20:49,048 - INFO - Extracted: c8ebcb57_PNADC_032024.txt
2026-10-08 10:20:49,050 - INFO - Processing (depth 0): PNADC_042024_20250815.zip
2026-10-08 10:21:01,331 - INFO - Extracted: c1bffc2a_PNADC_042024.txt
Processing files: 100%|████████████████████████████████████████████████████████████| 4/4 [02:45<00:00, 41.41s/it]
2026-10-08 10:23:46,985 - INFO - Successfully processed 4/4 files
2026-10-08 10:23:46,985 - INFO - Extraction completed successfully.
bra_dfs[0].head()
| Ano | Trimestre | UF | Capital | RM_RIDE | UPA | Estrato | V1008 | V1014 | V1016 | ... | V1028192 | V1028193 | V1028194 | V1028195 | V1028196 | V1028197 | V1028198 | V1028199 | V1028200 | filename | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2024 | 1 | 11 | 11 | <NA> | 110000016 | 1110011 | 03 | 11 | 1 | ... | 000242.37393247 | 000000.00000000 | 000000.00000000 | 000132.86482247 | 000252.85458864 | 000271.03799675 | 000122.61081652 | 000125.78602243 | 000113.09511303 | a7db871d_PNADC_012024.txt |
| 1 | 2024 | 1 | 11 | 11 | <NA> | 110000016 | 1110011 | 06 | 11 | 1 | ... | 000405.66107457 | 000000.00000000 | 000000.00000000 | 000205.06572241 | 000410.23613176 | 000437.83686366 | 000190.08927267 | 000200.15696949 | 000182.15329508 | a7db871d_PNADC_012024.txt |
| 2 | 2024 | 1 | 11 | 11 | <NA> | 110000016 | 1110011 | 06 | 11 | 1 | ... | 000405.66107457 | 000000.00000000 | 000000.00000000 | 000205.06572241 | 000410.23613176 | 000437.83686366 | 000190.08927267 | 000200.15696949 | 000182.15329508 | a7db871d_PNADC_012024.txt |
| 3 | 2024 | 1 | 11 | 11 | <NA> | 110000016 | 1110011 | 08 | 11 | 1 | ... | 000485.38386591 | 000000.00000000 | 000000.00000000 | 000242.53160028 | 000474.75504741 | 000520.88948037 | 000223.17316781 | 000229.81795045 | 000213.22589782 | a7db871d_PNADC_012024.txt |
| 4 | 2024 | 1 | 11 | 11 | <NA> | 110000016 | 1110011 | 08 | 11 | 1 | ... | 000485.38386591 | 000000.00000000 | 000000.00000000 | 000242.53160028 | 000474.75504741 | 000520.88948037 | 000223.17316781 | 000229.81795045 | 000213.22589782 | a7db871d_PNADC_012024.txt |
5 rows × 421 columns
Use case 3: Extracting data from Peru#
Peruvian data is extracted from the National Institute of Statistics and Informatics (INEI) website. In this case, we are extracting the National Household Survey (ENAHO) for the year 2022. The down_ext parameter is set to download .csv and .zip files, and the sep parameter is set to ;, indicating that the data files are semicolon-separated values.
per_online_extractor = Extractor(input_path="https://www.inei.gob.pe/media/DATOS_ABIERTOS/ENAHO/DATA/2022.zip", down_ext=['.csv','.zip'], output_path="../data", depth=0)
per_dfs = per_online_extractor.s4h_extract()
2026-10-08 10:25:33,693 - INFO - ----------------------
2026-10-08 10:25:33,693 - INFO - Starting data extraction...
2026-10-08 10:25:33,694 - INFO - Extracting data in online mode...
2026-10-08 10:25:33,694 - INFO - Detected direct file download URL - skipping scraping
2026-10-08 10:25:33,695 - INFO - Downloading large file (2022.zip)...
0.00B [00:00, ?B/s]2026-10-08 10:25:51,275 - INFO - Successfully downloaded: 2022.zip
0.00B [00:17, ?B/s]
2026-10-08 10:25:51,277 - INFO - Processing (depth 0): 2022.zip
2026-10-08 10:26:06,472 - INFO - Extracted: a0643d91_784-Modulo01_Enaho01-2022-100.csv
2026-10-08 10:26:06,474 - INFO - Extracted: a0643d91_784-Modulo02_ENAHO-TABLA-CIUO-88.csv
2026-10-08 10:26:06,475 - INFO - Extracted: a0643d91_784-Modulo02_ENAHO-TABLA-CNO-2015.csv
2026-10-08 10:26:07,401 - INFO - Extracted: a0643d91_784-Modulo02_Enaho01-2022-200.csv
2026-10-08 10:26:15,361 - INFO - Extracted: a0643d91_784-Modulo03_Enaho01a-2022-300.csv
2026-10-08 10:26:15,365 - INFO - Extracted: a0643d91_784-Modulo04_ENAHO-TABLA-PAISES.csv
2026-10-08 10:26:15,367 - INFO - Extracted: a0643d91_784-Modulo04_ENAHO-TABLA-UBIGEO.csv
2026-10-08 10:26:16,355 - INFO - Extracted: a0643d91_784-Modulo04_Enaho01a-2022-400.csv
2026-10-08 10:26:16,358 - INFO - Extracted: a0643d91_784-Modulo04_TABLA-UBIGEO-1874 DISTRITOS.csv
2026-10-08 10:26:16,361 - INFO - Extracted: a0643d91_784-Modulo05_ENAHO-TABLA-CIIU-REV3.csv
2026-10-08 10:26:16,363 - INFO - Extracted: a0643d91_784-Modulo05_ENAHO-TABLA-CIIU-REV4.csv
2026-10-08 10:26:16,365 - INFO - Extracted: a0643d91_784-Modulo05_ENAHO-TABLA-CIUO-88.csv
2026-10-08 10:26:16,366 - INFO - Extracted: a0643d91_784-Modulo05_ENAHO-TABLA-CNO-2015.csv
2026-10-08 10:26:17,296 - INFO - Extracted: a0643d91_784-Modulo05_Enaho01a-2022-500.csv
2026-10-08 10:26:18,565 - INFO - Extracted: a0643d91_784-Modulo07_Enaho01-2022-601.csv
2026-10-08 10:26:18,583 - INFO - Extracted: a0643d91_784-Modulo08_Enaho01-2022-602.csv
2026-10-08 10:26:18,587 - INFO - Extracted: a0643d91_784-Modulo08_Enaho01-2022-602A.csv
2026-10-08 10:26:18,630 - INFO - Extracted: a0643d91_784-Modulo09_Enaho01-2022-603.csv
2026-10-08 10:26:18,648 - INFO - Extracted: a0643d91_784-Modulo10_Enaho01-2022-604.csv
2026-10-08 10:26:18,657 - INFO - Extracted: a0643d91_784-Modulo11_Enaho01-2022-605.csv
2026-10-08 10:26:18,670 - INFO - Extracted: a0643d91_784-Modulo12_Enaho01-2022-606.csv
2026-10-08 10:26:18,680 - INFO - Extracted: a0643d91_784-Modulo13_Enaho01-2022-607.csv
2026-10-08 10:26:18,688 - INFO - Extracted: a0643d91_784-Modulo15_Enaho01-2022-609.csv
2026-10-08 10:26:18,697 - INFO - Extracted: a0643d91_784-Modulo16_Enaho01-2022-610.csv
2026-10-08 10:26:18,713 - INFO - Extracted: a0643d91_784-Modulo17_Enaho01-2022-611.csv
2026-10-08 10:26:18,738 - INFO - Extracted: a0643d91_784-Modulo18_Enaho01-2022-612.csv
2026-10-08 10:26:18,740 - INFO - Extracted: a0643d91_784-Modulo22_ENAHO-TABLA-AGROPECUARIO.csv
2026-10-08 10:26:18,744 - INFO - Extracted: a0643d91_784-Modulo22_Enaho02-2022-2000.csv
2026-10-08 10:26:18,745 - INFO - Extracted: a0643d91_784-Modulo22_Enaho02-2022-2000A.csv
2026-10-08 10:26:18,750 - INFO - Extracted: a0643d91_784-Modulo22_Enaho02-2022-2100.csv
2026-10-08 10:26:18,751 - INFO - Extracted: a0643d91_784-Modulo23_ENAHO-TABLA-AGROPECUARIO.csv
2026-10-08 10:26:18,753 - INFO - Extracted: a0643d91_784-Modulo23_Enaho02-2022-2200.csv
2026-10-08 10:26:18,755 - INFO - Extracted: a0643d91_784-Modulo24_ENAHO-TABLA-AGROPECUARIO.csv
2026-10-08 10:26:18,756 - INFO - Extracted: a0643d91_784-Modulo24_Enaho02-2022-2300.csv
2026-10-08 10:26:18,757 - INFO - Extracted: a0643d91_784-Modulo25_ENAHO-TABLA-AGROPECUARIO.csv
2026-10-08 10:26:18,759 - INFO - Extracted: a0643d91_784-Modulo25_Enaho02-2022-2400.csv
2026-10-08 10:26:18,761 - INFO - Extracted: a0643d91_784-Modulo26_ENAHO-TABLA-AGROPECUARIO.csv
2026-10-08 10:26:18,764 - INFO - Extracted: a0643d91_784-Modulo26_Enaho02-2022-2500.csv
2026-10-08 10:26:18,766 - INFO - Extracted: a0643d91_784-Modulo27_ENAHO-TABLA-AGROPECUARIO.csv
2026-10-08 10:26:18,768 - INFO - Extracted: a0643d91_784-Modulo27_Enaho02-2022-2600.csv
2026-10-08 10:26:18,770 - INFO - Extracted: a0643d91_784-Modulo28_ENAHO-TABLA-AGROPECUARIO.csv
2026-10-08 10:26:18,771 - INFO - Extracted: a0643d91_784-Modulo28_Enaho02-2022-2700.csv
2026-10-08 10:26:18,781 - INFO - Extracted: a0643d91_784-Modulo34_Sumaria-2022-12g.csv
2026-10-08 10:26:18,789 - INFO - Extracted: a0643d91_784-Modulo34_Sumaria-2022.csv
2026-10-08 10:26:18,792 - INFO - Extracted: a0643d91_784-Modulo37_Enaho01-2022-700.csv
2026-10-08 10:26:18,795 - INFO - Extracted: a0643d91_784-Modulo37_Enaho01-2022-700A.csv
2026-10-08 10:26:18,799 - INFO - Extracted: a0643d91_784-Modulo37_Enaho01-2022-700B.csv
2026-10-08 10:26:18,802 - INFO - Extracted: a0643d91_784-Modulo77_Enaho04-2022-1-Preg-1-a-13.csv
2026-10-08 10:26:18,805 - INFO - Extracted: a0643d91_784-Modulo77_Enaho04-2022-2-Preg-14-a-22.csv
2026-10-08 10:26:18,814 - INFO - Extracted: a0643d91_784-Modulo77_Enaho04-2022-3-Preg-23.csv
2026-10-08 10:26:18,817 - INFO - Extracted: a0643d91_784-Modulo77_Enaho04-2022-4-Preg-24.csv
2026-10-08 10:26:18,820 - INFO - Extracted: a0643d91_784-Modulo77_Enaho04-2022-5-Preg-25.csv
2026-10-08 10:26:18,836 - INFO - Extracted: a0643d91_784-Modulo78_Enaho01-2022-606D.csv
2026-10-08 10:26:18,839 - INFO - Extracted: a0643d91_784-Modulo84_Enaho01-2022-800A.csv
2026-10-08 10:26:18,841 - INFO - Extracted: a0643d91_784-Modulo84_Enaho01-2022-800B.csv
2026-10-08 10:26:18,848 - INFO - Extracted: a0643d91_784-Modulo85_Enaho01B-2022-1.csv
2026-10-08 10:26:18,851 - INFO - Extracted: a0643d91_784-Modulo85_Enaho01B-2022-2.csv
Processing files: 100%|██████████████████████████████████████████████████████████| 57/57 [00:30<00:00, 1.86it/s]
2026-10-08 10:26:49,616 - INFO - Successfully processed 57/57 files
2026-10-08 10:26:49,616 - INFO - Extraction completed successfully.
per_dfs[0].head()
| GRUPO | SUBGRUPO | CODIGO | PRODUCTO | filename | |
|---|---|---|---|---|---|
| 0 | 10 | 0 | 1000 | PRODUCTOS | a0643d91_784-Modulo27_ENAHO-TABLA-AGROPECUARIO... |
| 1 | 11 | 0 | 1100 | FRUTALES | a0643d91_784-Modulo27_ENAHO-TABLA-AGROPECUARIO... |
| 2 | 11 | 1 | 1101 | AGUAJE | a0643d91_784-Modulo27_ENAHO-TABLA-AGROPECUARIO... |
| 3 | 11 | 2 | 1102 | ALBARICOQUE O DAMASCO | a0643d91_784-Modulo27_ENAHO-TABLA-AGROPECUARIO... |
| 4 | 11 | 3 | 1103 | ALMENDRO | a0643d91_784-Modulo27_ENAHO-TABLA-AGROPECUARIO... |
Further steps#
Harmonize the extracted data using the
Harmonizerclass from thesocio4healthlibrary. You can follow the Harmonization tutorial for more details.