Hands-on with socio4health: socioeconomic and demographic variables on dengue incidence in Colombia#
Run the tutorial via free cloud platforms: 
This notebook provides you with a real world example on how to use socio4health to retrieve, harmonize and analyze socioeconomic and demographic variables related to dengue incidence in Colombia and recreate the dataset used in the publication Exploring Dengue Dynamics: A Multi-Scale Analysis of Spatio-Temporal Trends in Ibagué, Colombia by Otelo et al., published in Virus in 2024 (DOI). This tutorial assumes you have an intermediate or advanced understanding of Python and data manipulation.
Setting up the environment#
To run this notebook, you need to have the following prerequisites:
Python 3.10+
Additionally, you need to install the socio4health and pandas package, which can be done using pip:
!pip install socio4health pandas -q
[notice] A new release of pip is available: 25.1.1 -> 25.2
[notice] To update, run: python.exe -m pip install --upgrade pip
%pip install "scrapy==2.14.1"
import scrapy
print(scrapy.__version__)
Requirement already satisfied: scrapy==2.14.1 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (2.14.1)
Requirement already satisfied: cryptography>=37.0.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (46.0.5)
Requirement already satisfied: cssselect>=0.9.1 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (1.4.0)
Requirement already satisfied: defusedxml>=0.7.1 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (0.7.1)
Requirement already satisfied: itemadapter>=0.1.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (0.13.1)
Requirement already satisfied: itemloaders>=1.0.1 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (1.4.0)
Requirement already satisfied: lxml>=4.6.4 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (6.0.2)
Requirement already satisfied: packaging in c:\users\equipo\appdata\roaming\python\python312\site-packages (from scrapy==2.14.1) (26.0)
Requirement already satisfied: parsel>=1.5.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (1.11.0)
Requirement already satisfied: protego>=0.1.15 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (0.6.0)
Requirement already satisfied: pydispatcher>=2.0.5 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (2.0.7)
Requirement already satisfied: pyopenssl>=22.0.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (25.3.0)
Requirement already satisfied: queuelib>=1.4.2 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (1.9.0)
Requirement already satisfied: service-identity>=18.1.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (24.2.0)
Requirement already satisfied: tldextract in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (5.3.1)
Requirement already satisfied: twisted<=25.5.0,>=21.7.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (25.5.0)
Requirement already satisfied: w3lib>=1.17.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (2.4.0)
Requirement already satisfied: zope-interface>=5.1.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from scrapy==2.14.1) (8.2)
Requirement already satisfied: cffi>=2.0.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from cryptography>=37.0.0->scrapy==2.14.1) (2.0.0)
Requirement already satisfied: jmespath>=0.9.5 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from itemloaders>=1.0.1->scrapy==2.14.1) (1.1.0)
Requirement already satisfied: typing-extensions>=4.9 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from pyopenssl>=22.0.0->scrapy==2.14.1) (4.15.0)
Requirement already satisfied: attrs>=19.1.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from service-identity>=18.1.0->scrapy==2.14.1) (25.4.0)
Requirement already satisfied: pyasn1 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from service-identity>=18.1.0->scrapy==2.14.1) (0.6.2)
Requirement already satisfied: pyasn1-modules in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from service-identity>=18.1.0->scrapy==2.14.1) (0.4.2)
Requirement already satisfied: automat>=24.8.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from twisted<=25.5.0,>=21.7.0->scrapy==2.14.1) (25.4.16)
Requirement already satisfied: constantly>=15.1 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from twisted<=25.5.0,>=21.7.0->scrapy==2.14.1) (23.10.4)
Requirement already satisfied: hyperlink>=17.1.1 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from twisted<=25.5.0,>=21.7.0->scrapy==2.14.1) (21.0.0)
Requirement already satisfied: incremental>=24.7.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from twisted<=25.5.0,>=21.7.0->scrapy==2.14.1) (24.11.0)
Requirement already satisfied: idna in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from tldextract->scrapy==2.14.1) (3.11)
Requirement already satisfied: requests>=2.1.0 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from tldextract->scrapy==2.14.1) (2.32.5)
Requirement already satisfied: requests-file>=1.4 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from tldextract->scrapy==2.14.1) (3.0.1)
Requirement already satisfied: filelock>=3.0.8 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from tldextract->scrapy==2.14.1) (3.21.2)
Requirement already satisfied: pycparser in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from cffi>=2.0.0->cryptography>=37.0.0->scrapy==2.14.1) (3.0)
Requirement already satisfied: charset_normalizer<4,>=2 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from requests>=2.1.0->tldextract->scrapy==2.14.1) (3.4.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from requests>=2.1.0->tldextract->scrapy==2.14.1) (2.6.3)
Requirement already satisfied: certifi>=2017.4.17 in f:\users\equipo\appdata\local\programs\python\python312\lib\site-packages (from requests>=2.1.0->tldextract->scrapy==2.14.1) (2026.1.4)
Note: you may need to restart the kernel to use updated packages.
2.14.1
[notice] A new release of pip is available: 24.2 -> 26.2.1
[notice] To update, run: F:\Users\EQUIPO\AppData\Local\Programs\Python\Python312\python.exe -m pip install --upgrade pip
Import Libraries#
To perform the data extraction, the socio4health library provides the Extractor class for data extraction, and the Harmonizer class for data harmonization of the retrieved date. pandas will be used for data manipulation. Additionally, we will use some utility functions from the socio4health.utils.harmonizer_utils module to standardize and translate the dictionary.
import datetime
import geopandas as gpd
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from socio4health import Extractor
from socio4health.harmonizer import Harmonizer
from socio4health.utils import harmonizer_utils
1. Extract data from Colombia#
To extract data from Colombia, use the Extractor class from the socio4health library. As in the publication, extract the Colombian National Population and Housing Census 2018 (CNPV 2018) dataset from the Colombian Nacional Administration of Statistics (DANE) website. The dataset is available at: https://microdatos.dane.gov.co/index.php/catalog/643/related-materials.
The Extractor class requires the following parameters:
input_path: TheURLor local path to the data source.down_ext: A list of file extensions to download. This can include.CSV,.csv,.zip, etc.output_path: The local path where the extracted data will be saved.key_words: A list of keywords to filter the files to be downloaded. In this case, we are only interested in the file14045.zip, which contains the data at the desired level of granularity (census block level or “Manzana”).depth: The depth of the directory structure to traverse when downloading files. A depth of0means only the files in the specified directory will be downloaded.
col_online_extractor = Extractor(input_path="https://microdatos.dane.gov.co/index.php/catalog/643/related-materials",
down_ext=['.cpg', '.dbf', '.prj','.sbn', '.sbx', '.shx', '.shp', '.zip'],
output_path="../CNVP2018",
key_words=["14045.zip"],
depth=0)
col_CNPV = col_online_extractor.s4h_extract()
2026-10-08 09:27:51,695 - INFO - ----------------------
2026-10-08 09:27:51,695 - INFO - Starting data extraction...
2026-10-08 09:27:51,696 - INFO - Extracting data in online mode...
2026-10-08 09:27:51,696 - INFO - Scraping URL: https://microdatos.dane.gov.co/index.php/catalog/643/related-materials with depth 0
2026-10-08 09:27:53,701 - INFO - Spider completed successfully for URL: https://microdatos.dane.gov.co/index.php/catalog/643/related-materials
2026-10-08 09:27:53,708 - INFO - Downloading files to: ../CNVP2018
Downloading files: 0%| | 0/1 [00:00<?, ?it/s]2026-10-08 09:28:02,069 - INFO - Successfully downloaded: 14045.zip
Downloading files: 100%|███████████████████████████████████████████████████████████| 1/1 [00:08<00:00, 8.36s/it]
2026-10-08 09:28:02,071 - INFO - Processing (depth 0): 14045.zip
2026-10-08 09:28:04,508 - INFO - Extracted: 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_ANM_MANZANA.cpg
2026-10-08 09:28:28,940 - INFO - Extracted: 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_ANM_MANZANA.dbf
2026-10-08 09:28:28,942 - INFO - Extracted: 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_ANM_MANZANA.prj
2026-10-08 09:28:28,947 - INFO - Extracted: 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_ANM_MANZANA.sbn
2026-10-08 09:28:28,949 - INFO - Extracted: 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_ANM_MANZANA.sbx
2026-10-08 09:28:33,763 - INFO - Extracted: 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_ANM_MANZANA.shp
2026-10-08 09:28:33,766 - INFO - Extracted: 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_ANM_MANZANA.shx
Processing files: 0%| | 0/7 [00:00<?, ?it/s]2026-10-08 09:28:33,768 - WARNING - Unsupported extension: .prj
2026-10-08 09:28:33,769 - WARNING - Unsupported extension: .shx
2026-10-08 09:28:33,769 - WARNING - Unsupported extension: .sbn
2026-10-08 09:28:33,769 - WARNING - Unsupported extension: .sbx
2026-10-08 09:28:33,770 - WARNING - Unsupported extension: .dbf
Processing files: 86%|███████████████████████████████████████████████████▍ | 6/7 [00:20<00:03, 3.41s/it]2026-10-08 09:28:54,245 - WARNING - Unsupported extension: .cpg
Processing files: 100%|████████████████████████████████████████████████████████████| 7/7 [00:20<00:00, 2.93s/it]
2026-10-08 09:28:54,246 - INFO - Successfully processed 7/7 files
2026-10-08 09:28:54,246 - INFO - Extraction completed successfully.
col_CNPV[0]
| COD_DANE_A | DPTO_CCDGO | MPIO_CCDGO | MPIO_CDPMP | CLAS_CCDGO | SETR_CCDGO | SETR_CCNCT | SECR_CCDGO | SECR_CCNCT | ZU_CCDGO | ... | TP51_13_ED | TP51_99_ED | CD_LC_CM | NMB_LC_CM | TP_LC_CM | Shape_Leng | Shape_Area | COD_RDTM | geometry | filename | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0500210000000000010101 | 05 | 002 | 05002 | 1 | 000 | 050021000 | 00 | 05002100000 | 000 | ... | 10.0 | 1.0 | NaN | NaN | NaN | 0.002298 | 2.038760e-07 | 050021990000000000010101 | POLYGON ((-75.42779 5.79423, -75.4278 5.79422,... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
| 1 | 0500210000000000010102 | 05 | 002 | 05002 | 1 | 000 | 050021000 | 00 | 05002100000 | 000 | ... | 19.0 | 4.0 | NaN | NaN | NaN | 0.003402 | 5.600867e-07 | 050021990000000000010102 | POLYGON ((-75.42719 5.79421, -75.42715 5.79415... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
| 2 | 0500210000000000010103 | 05 | 002 | 05002 | 1 | 000 | 050021000 | 00 | 05002100000 | 000 | ... | 6.0 | 1.0 | NaN | NaN | NaN | 0.002622 | 4.293780e-07 | 050021990000000000010103 | POLYGON ((-75.42804 5.79294, -75.42807 5.79291... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
| 3 | 0500210000000000010104 | 05 | 002 | 05002 | 1 | 000 | 050021000 | 00 | 05002100000 | 000 | ... | 11.0 | 2.0 | NaN | NaN | NaN | 0.002673 | 4.493171e-07 | 050021990000000000010104 | POLYGON ((-75.42853 5.79348, -75.4286 5.79342,... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
| 4 | 0500210000000000010105 | 05 | 002 | 05002 | 1 | 000 | 050021000 | 00 | 05002100000 | 000 | ... | 0.0 | 0.0 | NaN | NaN | NaN | 0.001338 | 8.776894e-08 | 050021990000000000010105 | POLYGON ((-75.4291 5.79393, -75.4291 5.79393, ... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 504991 | 9977320030102700010102 | 99 | 773 | 99773 | 2 | 003 | 997732003 | 01 | 99773200301 | 027 | ... | 0.0 | 0.0 | NaN | NaN | NaN | 0.002270 | 2.451366e-07 | 997732990030102700010102 | POLYGON ((-69.85155 4.33427, -69.85149 4.33427... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
| 504992 | 9977320030102700010103 | 99 | 773 | 99773 | 2 | 003 | 997732003 | 01 | 99773200301 | 027 | ... | 0.0 | 0.0 | NaN | NaN | NaN | 0.002919 | 5.196303e-07 | 997732990030102700010103 | POLYGON ((-69.85227 4.33365, -69.85257 4.3337,... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
| 504993 | 9977320030102700010104 | 99 | 773 | 99773 | 2 | 003 | 997732003 | 01 | 99773200301 | 027 | ... | 0.0 | 0.0 | NaN | NaN | NaN | 0.002938 | 3.280837e-07 | 997732990030102700010104 | POLYGON ((-69.85276 4.33338, -69.85274 4.3336,... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
| 504994 | 9977320030102700010105 | 99 | 773 | 99773 | 2 | 003 | 997732003 | 01 | 99773200301 | 027 | ... | 0.0 | 0.0 | NaN | NaN | NaN | 0.002514 | 3.375903e-07 | 997732990030102700010105 | POLYGON ((-69.85313 4.33348, -69.85311 4.33368... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
| 504995 | 9977320030102700010106 | 99 | 773 | 99773 | 2 | 003 | 997732003 | 01 | 99773200301 | 027 | ... | 0.0 | 0.0 | NaN | NaN | NaN | 0.002535 | 1.992453e-07 | 997732990030102700010106 | POLYGON ((-69.85425 4.33397, -69.8541 4.33396,... | 383d6920_MGN_NivelManzana_Integrado_CNPV_MGN_A... |
504996 rows × 111 columns
2. Load and standardize the dictionary#
To harmonize the data, provide a dictionary that describes the variables in the dataset. The CNPV 2018 dictionary is available at here. Follow the steps in the tutorial “How to Create a Raw Dictionary for Data Harmonization” to create a raw dictionary in Excel format.
raw_dic = pd.read_excel("raw_dic_mzn.xlsx")
raw_dic
| variable_name | type | size | question | description | value | |
|---|---|---|---|---|---|---|
| 0 | COD_DANE_A | Text | 22.0 | Código de manzana concatenado (departamento, m... | NaN | NaN |
| 1 | DPTO_CCDGO | Text | 2.0 | Código del departamento | NaN | NaN |
| 2 | MPIO_CCDGO | Text | 3.0 | Código del municipio | NaN | NaN |
| 3 | MPIO_CDPMP | Text | 5.0 | Código concatenado que identifica al municipio | NaN | NaN |
| 4 | CLAS_CCDGO | Text | 1.0 | Código de la clase 1 cabecera municipal, 2 cen... | NaN | NaN |
| ... | ... | ... | ... | ... | ... | ... |
| 101 | TP51_13_ED | Double | NaN | Conteo de personas donde el nivel educativo de... | Ninguno | NaN |
| 102 | TP51_99_ED | Double | NaN | Conteo de personas donde el nivel educativo de... | Sin información | NaN |
| 103 | CD_LC_CM | Text | 10.0 | Código de la localidad o comuna | NaN | NaN |
| 104 | NMB_LC_CM | Text | 50.0 | Nombre de la localidad o comuna | NaN | NaN |
| 105 | TP_LC_CM | Text | 20.0 | Descripción de tipo localidad o comuna o corre... | NaN | NaN |
106 rows × 6 columns
Standardize the dictionary using the s4h_standardize_dict function from the harmonizer_utils module. Then, translate it to English using the s4h_translate_column function from the same module (Note: The translation model is downloaded on first use and cached locally, so the first execution may take a few minutes). Finally, classify the variables into categories using a pre-trained BERT model.
dic = harmonizer_utils.s4h_standardize_dict(raw_dic)
dic = harmonizer_utils.s4h_translate_column(dic, "question", language="en")
dic = harmonizer_utils.s4h_translate_column(dic, "description", language="en")
dic = harmonizer_utils.s4h_translate_column(dic, "possible_answers", language="en")
2026-10-08 09:32:05,842 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-10-08 09:32:05,843 - WARNING - Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-10-08 09:32:05,861 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/config.json "HTTP/1.1 200 OK"
2026-10-08 09:32:05,972 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-08 09:32:05,995 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/tokenizer_config.json "HTTP/1.1 200 OK"
2026-10-08 09:32:06,099 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-10-08 09:32:06,200 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
F:\Users\EQUIPO\AppData\Local\Programs\Python\Python312\Lib\site-packages\transformers\models\marian\tokenization_marian.py:176: UserWarning: Recommended: pip install sacremoses.
warnings.warn("Recommended: pip install sacremoses.")
2026-10-08 09:32:06,394 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-08 09:32:06,425 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/config.json "HTTP/1.1 200 OK"
2026-10-08 09:32:06,909 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/model.safetensors "HTTP/1.1 404 Not Found"
2026-10-08 09:32:07,036 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en "HTTP/1.1 200 OK"
2026-10-08 09:32:07,285 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/commits/main "HTTP/1.1 200 OK"
2026-10-08 09:32:07,663 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/discussions?p=0 "HTTP/1.1 200 OK"
2026-10-08 09:32:08,071 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/commits/refs%2Fpr%2F6 "HTTP/1.1 200 OK"
The tied weights mapping and config for this model specifies to tie model.shared.weight to model.decoder.embed_tokens.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning
The tied weights mapping and config for this model specifies to tie model.shared.weight to model.encoder.embed_tokens.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning
2026-10-08 09:32:08,323 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/refs%2Fpr%2F6/model.safetensors.index.json "HTTP/1.1 404 Not Found"
2026-10-08 09:32:08,376 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-08 09:32:08,399 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/generation_config.json "HTTP/1.1 200 OK"
2026-10-08 09:32:08,430 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/refs%2Fpr%2F6/model.safetensors "HTTP/1.1 302 Found"
question translated
description translated
possible_answers translated
dic = harmonizer_utils.s4h_classify_rows(dic, "question_en", "description_en", "possible_answers_en",
new_column_name="category",
MODEL_PATH="dirreno/harmonize_bert_finetuned_classifier")
2026-10-08 09:34:33,599 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-08 09:34:33,615 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/config.json "HTTP/1.1 200 OK"
2026-10-08 09:34:33,952 - INFO - HTTP Request: GET https://huggingface.co/api/models/dirreno/harmonize_bert_finetuned_classifier/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-10-08 09:34:34,067 - INFO - HTTP Request: GET https://huggingface.co/api/models/dirreno/harmonize_bert_finetuned_classifier/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
This is how the standardized and translated dictionary looks:
dic
| type | size | description | value | possible_answers | question | variable_name | question_en | description_en | possible_answers_en | category | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Long Integer | NaN | NaN | NaN | NaN | año de la información geográfica | VERSION | year of geographical information | NaN | NaN | Identification |
| 1 | Double | NaN | NaN | NaN | NaN | cantidad de encuestas cnpv 2018 | CTNENCUEST | number of surveys cnpv 2018 | NaN | NaN | Identification |
| 2 | Double | NaN | NaN | NaN | NaN | cantidad de encuestas que reportaron estar en ... | TP3_1_SI | number of surveys reported to be in ethnic ter... | NaN | NaN | Identification |
| 3 | Double | NaN | resguardo indígena | NaN | NaN | cantidad de encuestas que reportaron estar en ... | TP3A_RI | number of surveys reported to be in ethnic ind... | protection of indigenous peoples | NaN | Identification |
| 4 | Double | NaN | tccn | NaN | NaN | cantidad de encuestas que reportaron estar en ... | TP3B_TCN | number of surveys reported to be in ethnic ter... | ccn | NaN | Identification |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 101 | Text | 50.0 | NaN | NaN | NaN | nombre capa anonimizada | DATO_ANM | anonymized layer name | NaN | NaN | Identification |
| 102 | Text | 50.0 | NaN | NaN | NaN | nombre de la localidad o comuna | NMB_LC_CM | name of town or commune | NaN | NaN | Business |
| 103 | Double | NaN | NaN | NaN | NaN | número de personas | TP27_PERSO | number of persons | NaN | NaN | Identification |
| 104 | Double | NaN | NaN | NaN | NaN | número promedio de habitantes en la manzana qu... | DENSIDAD | average number of inhabitants in the block liv... | NaN | NaN | Housing |
| 105 | Double | NaN | NaN | NaN | NaN | área de la manzana en metros cuadrados (sistem... | AREA | area of the apple in square meters (system of ... | NaN | NaN | Identification |
106 rows × 11 columns
3. Harmonize the data#
Use the Harmonizer class from the socio4health library to harmonize the data. First, set the similarity threshold to 0.9, meaning that only variables with a similarity score of 0.9 or higher will be considered for harmonization. Next, set the nan_threshold to 1, meaning that only variables with no missing values will be considered for harmonization.
The available columns in the dataset can be checked using the s4h_get_available_columns method. This method takes a list of DataFrames as input and returns a list of column names that are present in the DataFrames.
har = Harmonizer()
har.similarity_threshold = 0.9
har.nan_threshold = 1
available_columns = har.s4h_get_available_columns(col_CNPV)
available_columns
['AG_CCDGO',
'AREA',
'CD_LC_CM',
'CLAS_CCDGO',
'COD_DANE_A',
'COD_RDTM',
'CTNENCUEST',
'DATO_ANM',
'DENSIDAD',
'DPTO_CCDGO',
'FILENAME',
'GEOMETRY',
'LATITUD',
'LONGITUD',
'MANZ_CCDGO',
'MPIO_CCDGO',
'MPIO_CDPMP',
'NMB_LC_CM',
'PERSONAS_L',
'PERSONAS_S',
'SECR_CCDGO',
'SECR_CCNCT',
'SECU_CCDGO',
'SECU_CCNCT',
'SETR_CCDGO',
'SETR_CCNCT',
'SETU_CCDGO',
'SETU_CCNCT',
'SHAPE_AREA',
'SHAPE_LENG',
'TP14_1_TIP',
'TP14_2_TIP',
'TP14_3_TIP',
'TP14_4_TIP',
'TP14_5_TIP',
'TP14_6_TIP',
'TP15_1_OCU',
'TP15_2_OCU',
'TP15_3_OCU',
'TP15_4_OCU',
'TP16_HOG',
'TP19_ACU_1',
'TP19_ACU_2',
'TP19_ALC_1',
'TP19_ALC_2',
'TP19_EE_1',
'TP19_EE_2',
'TP19_EE_E1',
'TP19_EE_E2',
'TP19_EE_E3',
'TP19_EE_E4',
'TP19_EE_E5',
'TP19_EE_E6',
'TP19_EE_E9',
'TP19_GAS_1',
'TP19_GAS_2',
'TP19_GAS_9',
'TP19_INTE1',
'TP19_INTE2',
'TP19_INTE9',
'TP19_RECB1',
'TP19_RECB2',
'TP27_PERSO',
'TP32_1_SEX',
'TP32_2_SEX',
'TP34_1_EDA',
'TP34_2_EDA',
'TP34_3_EDA',
'TP34_4_EDA',
'TP34_5_EDA',
'TP34_6_EDA',
'TP34_7_EDA',
'TP34_8_EDA',
'TP34_9_EDA',
'TP3A_RI',
'TP3B_TCN',
'TP3_1_SI',
'TP3_2_NO',
'TP4_1_SI',
'TP4_2_NO',
'TP51POSTGR',
'TP51PRIMAR',
'TP51SECUND',
'TP51SUPERI',
'TP51_13_ED',
'TP51_99_ED',
'TP9_1_USO',
'TP9_2_1_MI',
'TP9_2_2_MI',
'TP9_2_3_MI',
'TP9_2_4_MI',
'TP9_2_9_MI',
'TP9_2_USO',
'TP9_3_10_N',
'TP9_3_1_NO',
'TP9_3_2_NO',
'TP9_3_3_NO',
'TP9_3_4_NO',
'TP9_3_5_NO',
'TP9_3_6_NO',
'TP9_3_7_NO',
'TP9_3_8_NO',
'TP9_3_99_N',
'TP9_3_9_NO',
'TP9_3_USO',
'TP9_4_USO',
'TP_LC_CM',
'TVIVIENDA',
'VERSION',
'ZU_CCDGO',
'ZU_CDIVI']
Configure the dictionary, categories, additional columns, key column, and key values to prepare for the harmonization process. Set the dict_df parameter to the standardized and translated dictionary. Set the categories parameter to “Housing” to ensure that only housing-related variables are considered for harmonization. Set the extra_cols and key_colparameter to MPIO_CDPMP, which stands for the municipality code. This allows for filtering the data to include only records from a specific municipality. In this case, we are interested in the municipality with the code 73001, which corresponds to Ibague. Finally, use the s4h_data_selector method to filter the data based on the specified key column and key values. This method returns a list of DataFrames that match the filtering criteria.
col_CNPV[0].shape
(504996, 111)
har.dict_df = dic
har.categories = ["Housing"]
har.extra_cols = ['MPIO_CDPMP', 'GEOMETRY']
har.key_col = 'MPIO_CDPMP'
har.key_val = ['73001']
filtered_ddfs = har.s4h_data_selector(col_CNPV)
The method s4h_compare_with_dict can be used to compare the available columns in the dataset with the variables in the dictionary. This method helps to identify which variables from the dictionary are present in the dataset and can be harmonized.
Finally, display the filtered DataFrames to see the harmonized data for the specified municipality. If needed, the harmonized data can be exported to a CSV file using the to_csv method.
har.s4h_compare_with_dict(col_CNPV)
Matches with dict_df: 106 (95.50%)
| Unmatched ddfs variable | Unmatched dict_df variables | |
|---|---|---|
| 0 | COD_RDTM | None |
| 1 | FILENAME | None |
| 2 | GEOMETRY | None |
| 3 | SHAPE_AREA | None |
| 4 | SHAPE_LENG | None |
Finally, display the filtered DataFrames to see the harmonized data for the specified municipality. If needed, the harmonized data can be exported to a CSV file using the to_csv method or visualized using matplotlib, geopandas, and numpy as shown below:
filtered_ddfs
[ MPIO_CDPMP TP9_2_USO TP9_3_USO TP9_1_USO TP9_2_9_MI TP9_3_5_NO \
422086 73001 1.0 1.0 48.0 0.0 0.0
422087 73001 0.0 2.0 67.0 0.0 0.0
422088 73001 0.0 0.0 0.0 0.0 0.0
422089 73001 2.0 3.0 110.0 0.0 0.0
422090 73001 0.0 1.0 0.0 0.0 1.0
... ... ... ... ... ... ...
427978 73001 1.0 0.0 9.0 0.0 0.0
427979 73001 1.0 1.0 3.0 0.0 0.0
427980 73001 1.0 0.0 9.0 0.0 0.0
427981 73001 0.0 0.0 8.0 0.0 0.0
427982 73001 0.0 0.0 1.0 0.0 0.0
TP9_3_6_NO TP9_3_8_NO TP9_3_7_NO TP9_3_9_NO ... TP19_INTE2 \
422086 0.0 0.0 0.0 0.0 ... 38.0
422087 1.0 0.0 0.0 0.0 ... 8.0
422088 0.0 0.0 0.0 0.0 ... 0.0
422089 0.0 0.0 0.0 0.0 ... 63.0
422090 0.0 0.0 0.0 0.0 ... 0.0
... ... ... ... ... ... ...
427978 0.0 0.0 0.0 0.0 ... 5.0
427979 1.0 0.0 0.0 0.0 ... 0.0
427980 0.0 0.0 0.0 0.0 ... 8.0
427981 0.0 0.0 0.0 0.0 ... 7.0
427982 0.0 0.0 0.0 0.0 ... 0.0
TP14_2_TIP TP14_1_TIP TP14_6_TIP TP14_3_TIP TP14_4_TIP \
422086 10.0 22.0 1.0 16.0 0.0
422087 57.0 10.0 0.0 0.0 0.0
422088 0.0 0.0 0.0 0.0 0.0
422089 81.0 20.0 0.0 11.0 0.0
422090 0.0 0.0 0.0 0.0 0.0
... ... ... ... ... ...
427978 1.0 9.0 0.0 0.0 0.0
427979 0.0 4.0 0.0 0.0 0.0
427980 0.0 10.0 0.0 0.0 0.0
427981 0.0 8.0 0.0 0.0 0.0
427982 0.0 1.0 0.0 0.0 0.0
TP14_5_TIP TP15_3_OCU DENSIDAD \
422086 0.0 1.0 0.024806
422087 0.0 2.0 0.061506
422088 0.0 0.0 0.000000
422089 0.0 1.0 0.052900
422090 0.0 0.0 0.000000
... ... ... ...
427978 0.0 2.0 0.000233
427979 0.0 1.0 0.000000
427980 0.0 2.0 0.001260
427981 0.0 1.0 0.000529
427982 0.0 1.0 0.000000
GEOMETRY
422086 POLYGON ((-75.25204 4.45339, -75.25203 4.45339...
422087 POLYGON ((-75.24798 4.44987, -75.24797 4.44985...
422088 POLYGON ((-75.24873 4.44883, -75.24877 4.44879...
422089 POLYGON ((-75.24897 4.44924, -75.24893 4.44919...
422090 POLYGON ((-75.25099 4.45195, -75.25111 4.45187...
... ...
427978 POLYGON ((-75.19101 4.35893, -75.191 4.35893, ...
427979 POLYGON ((-75.18923 4.35806, -75.18929 4.35799...
427980 POLYGON ((-75.19102 4.35539, -75.19116 4.35513...
427981 POLYGON ((-75.19008 4.35431, -75.18991 4.35423...
427982 POLYGON ((-75.19121 4.35258, -75.19168 4.35285...
[5897 rows x 36 columns]]
df = filtered_ddfs[0]
# Create a GeoDataFrame
gdf = gpd.GeoDataFrame(df, geometry='GEOMETRY', crs="EPSG:4326")
gdf['TP19_GAS_1_Log'] =np.log(gdf['TP19_GAS_1'])
F:\Users\EQUIPO\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:402: RuntimeWarning: divide by zero encountered in log
result = getattr(ufunc, method)(*inputs, **kwargs)
# Plot
fig, ax = plt.subplots(figsize=(10, 8))
# plot without automatic colorbar
gdf.plot(column='TP19_GAS_1_Log', cmap='viridis', ax=ax)
# Plot with custom colorbar
sm = plt.cm.ScalarMappable(cmap='viridis')
sm.set_array(gdf['TP19_GAS_1_Log'])
cbar = fig.colorbar(sm, ax=ax, fraction=0.03, pad=0.02) # fraction controls the size of the colorbar
cbar.set_label("Gas line connection (log)")
ax.set_title("Gas line connection map", fontsize=14)
ax.set_axis_off()
plt.show()
F:\Users\EQUIPO\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\colors.py:2294: RuntimeWarning: invalid value encountered in subtract
resdat -= vmin
F:\Users\EQUIPO\AppData\Local\Programs\Python\Python312\Lib\site-packages\matplotlib\colors.py:2295: RuntimeWarning: invalid value encountered in divide
resdat /= (vmax - vmin)
