Hands-on with socio4health: effects of hydrometeorologigcal hazards and urbanization on dengue risk in Brazil#
Run the tutorial via free cloud platforms: 
This notebook provides a real-world example of how to use socio4health to retrieve, harmonize and analyze socioeconomic and demographic variables, such as the level of urbanization and access to water supply in Brazil, to recreate the dataset used in the publication Combined effects of hydrometeorological hazards and urbanisation on dengue risk in Brazil: a spatiotemporal modelling study by Lowe et al., published in The Lancet Planetary Health in 2021 (DOI). The study evaluated how the association between hydrometeorological events and dengue risk varies with these variables. This tutorial assumes an intermediate or advanced understanding of Python and data manipulation.
Setting up the environment#
To run this notebook, you need to have the following prerequisites:
Python 3.10+
Additionally, you need to install the socio4health and pandas package, which can be done using pip:
!pip install socio4health pandas -q
Import Libraries#
To perform the data extraction, the socio4health library provides the Extractor class for data extraction, and the Harmonizer class for data harmonization of the retrieved date. pandas will be used for data manipulation. Additionally, we will use some utility functions from the socio4health.utils.harmonizer_utils module to standardize and translate the dictionary.
import re
import pandas as pd
import dask.dataframe as dd
import matplotlib.pyplot as plt
from matplotlib.ticker import FuncFormatter
from socio4health import Extractor
from socio4health.harmonizer import Harmonizer
from socio4health.utils import harmonizer_utils, extractor_utils
1. Load and standardize the dictionary#
To harmonize the data, provide a dictionary that describes the variables in the dataset. The study retrieved data from the 2010 census, from Instituto Brasileiro de Geografia e Estatística (IBGE). The dictionary for the census data can be found here. Follow the steps in the tutorial “How to Create a Raw Dictionary for Data Harmonization” to create a raw dictionary in Excel format.
This dictionary must be standardized and translated to English. The socio4health.utils.harmonizer_utils module provides utility functions to perform these tasks. Additionally, the socio4health.utils.extractor_utils module provides utility functions to parse fixed-width file (FWF) dictionaries, which is the format used in the IBGE census data.
raw_dic = pd.read_excel("raw_dictionary_br_2010.xlsx")
dic=harmonizer_utils.s4h_standardize_dict(raw_dic)
colnames, colspecs =extractor_utils.s4h_parse_fwf_dict(dic)
This is how the standardized dictionary looks:
dic
| description | value | initial_position | final_position | size | dec | type | possible_answers | question | variable_name | |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | NaN | 1.0; 2.0; 9.0 | 107.0 | 107.0 | 1.0 | NaN | C | apenas um morador; mais de um morador; ignorado | a responsabilidade pelo domicílio é de: | V0402 |
| 1 | NaN | 1.0; 2.0; 3.0 | 90.0 | 90.0 | 1.0 | NaN | C | sim, em pelo menos um cômodo; sim, só na propr... | abastecimento de água, canalização: | V0209 |
| 2 | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0; 7.0; 8.0; 9.0; 10.0 | 88.0 | 89.0 | 2.0 | NaN | C | rede geral de distribuição; poço ou nascente n... | abastecimento de água, forma: | V0208 |
| 3 | NaN | 1.0; 2.0; 3.0 | 144.0 | 144.0 | 1.0 | NaN | C | adequada; semi-adequada; inadequada | adequação da moradia | V6210 |
| 4 | NaN | 1.0; 2.0 | 104.0 | 104.0 | 1.0 | NaN | C | sim; não | alguma pessoa que morava com você(s) estava mo... | V0301 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 71 | NaN | 1.0; 2.0 | 95.0 | 95.0 | 1.0 | NaN | C | sim; não | televisão, existência: | V0214 |
| 72 | NaN | 11.0; 12.0; 13.0; 14.0; 15.0; 51.0; 52.0; 53.0... | 56.0 | 57.0 | 2.0 | NaN | C\n | casa; casa de vila ou em condomínio; apartamen... | tipo de espécie: | V4002 |
| 73 | NaN | 11.0; 12.0; 13.0; 14.0; 15.0; 16.0; 17.0; 21.0... | 1.0 | 2.0 | 2.0 | NaN | A | rondônia; acre; amazonas; roraima; pará; amapá... | unidade da federação: | V0001 |
| 74 | NaN | NaN | 59.0 | 64.0 | 6.0 | NaN | N | NaN | valor do aluguel (em reais) | V2011 |
| 75 | NaN | NaN | 8.0 | 20.0 | 13.0 | NaN | A | NaN | área de ponderação | V0011 |
76 rows × 10 columns
The classification model used in this tutorial is a BERT model fine-tuned for the task of classifying survey questions into categories. You can use your own model by providing the path to the model in the MODEL_PATH parameter of the harmonizer_utils.s4h_classify_rows function.
dic = harmonizer_utils.s4h_translate_column(dic, "question", language="en")
dic = harmonizer_utils.s4h_translate_column(dic, "description", language="en")
dic = harmonizer_utils.s4h_translate_column(dic, "possible_answers", language="en")
dic = harmonizer_utils.s4h_classify_rows(dic, "question_en", "description_en", "possible_answers_en",
new_column_name="category",
MODEL_PATH="dirreno/harmonize_bert_finetuned_classifier")
dic
2026-10-06 14:38:04,909 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-06 14:38:04,924 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/config.json "HTTP/1.1 200 OK"
2026-10-06 14:38:05,024 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-10-06 14:38:05,025 - WARNING - Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-10-06 14:38:05,039 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/tokenizer_config.json "HTTP/1.1 200 OK"
2026-10-06 14:38:05,138 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-10-06 14:38:05,234 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
F:\Users\EQUIPO\AppData\Local\Programs\Python\Python312\Lib\site-packages\transformers\models\marian\tokenization_marian.py:176: UserWarning: Recommended: pip install sacremoses.
warnings.warn("Recommended: pip install sacremoses.")
2026-10-06 14:38:05,423 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-06 14:38:05,436 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/config.json "HTTP/1.1 200 OK"
2026-10-06 14:38:05,844 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/model.safetensors "HTTP/1.1 404 Not Found"
2026-10-06 14:38:05,963 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en "HTTP/1.1 200 OK"
2026-10-06 14:38:06,486 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/commits/main "HTTP/1.1 200 OK"
2026-10-06 14:38:06,592 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/discussions?p=0 "HTTP/1.1 200 OK"
2026-10-06 14:38:06,746 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/commits/refs%2Fpr%2F6 "HTTP/1.1 200 OK"
The tied weights mapping and config for this model specifies to tie model.shared.weight to model.decoder.embed_tokens.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning
The tied weights mapping and config for this model specifies to tie model.shared.weight to model.encoder.embed_tokens.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning
2026-10-06 14:38:07,316 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/refs%2Fpr%2F6/model.safetensors.index.json "HTTP/1.1 404 Not Found"
2026-10-06 14:38:07,335 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-06 14:38:07,351 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/generation_config.json "HTTP/1.1 200 OK"
2026-10-06 14:38:07,424 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/refs%2Fpr%2F6/model.safetensors "HTTP/1.1 302 Found"
question translated
description translated
possible_answers translated
2026-10-06 14:38:25,253 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/main/config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-06 14:38:25,268 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/config.json "HTTP/1.1 200 OK"
2026-10-06 14:38:25,562 - INFO - HTTP Request: GET https://huggingface.co/api/models/dirreno/harmonize_bert_finetuned_classifier/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-10-06 14:38:25,677 - INFO - HTTP Request: GET https://huggingface.co/api/models/dirreno/harmonize_bert_finetuned_classifier/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
| description | value | initial_position | final_position | size | dec | type | possible_answers | question | variable_name | question_en | description_en | possible_answers_en | category | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | NaN | 1.0; 2.0; 9.0 | 107.0 | 107.0 | 1.0 | NaN | C | apenas um morador; mais de um morador; ignorado | a responsabilidade pelo domicílio é de: | V0402 | the responsibility for the household is: | NaN | only one resident; more than one resident; ign... | Housing |
| 1 | NaN | 1.0; 2.0; 3.0 | 90.0 | 90.0 | 1.0 | NaN | C | sim, em pelo menos um cômodo; sim, só na propr... | abastecimento de água, canalização: | V0209 | water supply, pipeline: | NaN | yes, in at least one room; yes, only in proper... | Housing |
| 2 | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0; 7.0; 8.0; 9.0; 10.0 | 88.0 | 89.0 | 2.0 | NaN | C | rede geral de distribuição; poço ou nascente n... | abastecimento de água, forma: | V0208 | water supply, shape: | NaN | general distribution network; well or spring o... | Business |
| 3 | NaN | 1.0; 2.0; 3.0 | 144.0 | 144.0 | 1.0 | NaN | C | adequada; semi-adequada; inadequada | adequação da moradia | V6210 | adequacy of housing | NaN | adequate; semi-adequate; inadequate | Housing |
| 4 | NaN | 1.0; 2.0 | 104.0 | 104.0 | 1.0 | NaN | C | sim; não | alguma pessoa que morava com você(s) estava mo... | V0301 | someone who lived with you was living in anoth... | NaN | Yes; no | Business |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 71 | NaN | 1.0; 2.0 | 95.0 | 95.0 | 1.0 | NaN | C | sim; não | televisão, existência: | V0214 | television, existence: | NaN | Yes; no | Identification |
| 72 | NaN | 11.0; 12.0; 13.0; 14.0; 15.0; 51.0; 52.0; 53.0... | 56.0 | 57.0 | 2.0 | NaN | C\n | casa; casa de vila ou em condomínio; apartamen... | tipo de espécie: | V4002 | type of species: | NaN | house; town house or condominium; apartment; d... | Housing |
| 73 | NaN | 11.0; 12.0; 13.0; 14.0; 15.0; 16.0; 17.0; 21.0... | 1.0 | 2.0 | 2.0 | NaN | A | rondônia; acre; amazonas; roraima; pará; amapá... | unidade da federação: | V0001 | Federation unit: | NaN | rondônia; acre; amazon; roraima; pará; amapá; ... | Business |
| 74 | NaN | NaN | 59.0 | 64.0 | 6.0 | NaN | N | NaN | valor do aluguel (em reais) | V2011 | value of rent (in reals) | NaN | NaN | Business |
| 75 | NaN | NaN | 8.0 | 20.0 | 13.0 | NaN | A | NaN | área de ponderação | V0011 | weighting area | NaN | NaN | Housing |
76 rows × 14 columns
2. Extract data from Brazil Census 2010#
To extract data, use the Extractor class from the socio4health library. As in the publication, extract the Brazil Census 2010 dataset from the Brazilian Institute of Geography and Statistics (IBGE) website or from a local copy. The dataset is available here.
The Extractor class requires the following parameters:
input_path: TheURLor local path to the data source.down_ext: A list of file extensions to download. This can include.txt,.zip, etc.output_path: The local path where the extracted data will be saved.key_words: A list of keywords to filter the files to be downloaded. In this case, a regular expression is used to select only the files with a.zipextension that contain uppercase letters in their names.depth: The depth of the directory structure to traverse when downloading files. A depth of0means only the files in the specified directory will be downloaded.is_fwf: A boolean indicating whether the files are in fixed-width format (FWF). In this case, the files are in FWF format, so this parameter is set toTrue.colnames: A list of column names for the FWF files, extracted from the standardized dictionary.colspecs: A list of tuples indicating the start and end positions of each column in the FWF files, extracted from the standardized dictionary.
bra_online_extractor = Extractor(input_path="https://ftp.ibge.gov.br/Censos/Censo_Demografico_2010/Resultados_Gerais_da_Amostra/Microdados/",
down_ext=['.txt','.zip'],
output_path="IBGE_2010_",
key_words=[r"^[A-Z0-9_]+\.zip$"],
depth=0, is_fwf=True, colnames=colnames, colspecs=colspecs)
bra_Censo_2010 = bra_online_extractor.s4h_extract()
2026-10-06 14:38:28,420 - INFO - ----------------------
2026-10-06 14:38:28,420 - INFO - Starting data extraction...
2026-10-06 14:38:28,420 - INFO - Extracting data in online mode...
2026-10-06 14:38:28,421 - INFO - Scraping URL: https://ftp.ibge.gov.br/Censos/Censo_Demografico_2010/Resultados_Gerais_da_Amostra/Microdados/ with depth 0
2026-10-06 14:38:31,523 - INFO - Spider completed successfully for URL: https://ftp.ibge.gov.br/Censos/Censo_Demografico_2010/Resultados_Gerais_da_Amostra/Microdados/
2026-10-06 14:38:31,525 - INFO - Downloading files to: IBGE_2010_
Downloading files: 0%| | 0/28 [00:00<?, ?it/s]2026-10-06 14:38:33,824 - INFO - Successfully downloaded: AC.zip
Downloading files: 4%| | 1/28 [00:02<01:02, 2.30s/it]2026-10-06 14:38:36,689 - INFO - Successfully downloaded: AL.zip
Downloading files: 7%| | 2/28 [00:05<01:08, 2.63s/it]2026-10-06 14:38:39,996 - INFO - Successfully downloaded: AM.zip
Downloading files: 11%| | 3/28 [00:08<01:13, 2.94s/it]2026-10-06 14:38:42,202 - INFO - Successfully downloaded: AP.zip
Downloading files: 14%|▏| 4/28 [00:10<01:03, 2.65s/it]2026-10-06 14:38:49,304 - INFO - Successfully downloaded: BA.zip
Downloading files: 18%|▏| 5/28 [00:17<01:37, 4.26s/it]2026-10-06 14:38:53,744 - INFO - Successfully downloaded: CE.zip
Downloading files: 21%|▏| 6/28 [00:22<01:34, 4.32s/it]2026-10-06 14:38:56,261 - INFO - Successfully downloaded: DF.zip
Downloading files: 25%|▎| 7/28 [00:24<01:18, 3.73s/it]2026-10-06 14:38:59,794 - INFO - Successfully downloaded: ES.zip
Downloading files: 29%|▎| 8/28 [00:28<01:13, 3.67s/it]2026-10-06 14:39:04,260 - INFO - Successfully downloaded: GO.zip
Downloading files: 32%|▎| 9/28 [00:32<01:14, 3.92s/it]2026-10-06 14:39:08,595 - INFO - Successfully downloaded: MA.zip
Downloading files: 36%|▎| 10/28 [00:37<01:12, 4.05s/it2026-10-06 14:39:18,268 - INFO - Successfully downloaded: MG.zip
Downloading files: 39%|▍| 11/28 [00:46<01:38, 5.77s/it2026-10-06 14:39:21,067 - INFO - Successfully downloaded: MS.zip
Downloading files: 43%|▍| 12/28 [00:49<01:17, 4.86s/it2026-10-06 14:39:24,228 - INFO - Successfully downloaded: MT.zip
Downloading files: 46%|▍| 13/28 [00:52<01:05, 4.35s/it2026-10-06 14:39:28,202 - INFO - Successfully downloaded: PA.zip
Downloading files: 50%|▌| 14/28 [00:56<00:59, 4.24s/it2026-10-06 14:39:32,202 - INFO - Successfully downloaded: PB.zip
Downloading files: 54%|▌| 15/28 [01:00<00:54, 4.16s/it2026-10-06 14:39:37,320 - INFO - Successfully downloaded: PE.zip
Downloading files: 57%|▌| 16/28 [01:05<00:53, 4.45s/it2026-10-06 14:39:40,908 - INFO - Successfully downloaded: PI.zip
Downloading files: 61%|▌| 17/28 [01:09<00:46, 4.19s/it2026-10-06 14:39:47,050 - INFO - Successfully downloaded: PR.zip
Downloading files: 64%|▋| 18/28 [01:15<00:47, 4.78s/it2026-10-06 14:39:52,941 - INFO - Successfully downloaded: RJ.zip
Downloading files: 68%|▋| 19/28 [01:21<00:46, 5.11s/it2026-10-06 14:39:56,603 - INFO - Successfully downloaded: RN.zip
Downloading files: 71%|▋| 20/28 [01:25<00:37, 4.68s/it2026-10-06 14:39:59,753 - INFO - Successfully downloaded: RO.zip
Downloading files: 75%|▊| 21/28 [01:28<00:29, 4.22s/it2026-10-06 14:40:01,832 - INFO - Successfully downloaded: RR.zip
Downloading files: 79%|▊| 22/28 [01:30<00:21, 3.58s/it2026-10-06 14:40:08,281 - INFO - Successfully downloaded: RS.zip
Downloading files: 82%|▊| 23/28 [01:36<00:22, 4.44s/it2026-10-06 14:40:15,222 - INFO - Successfully downloaded: SC.zip
Downloading files: 86%|▊| 24/28 [01:43<00:20, 5.19s/it2026-10-06 14:40:18,038 - INFO - Successfully downloaded: SE.zip
Downloading files: 89%|▉| 25/28 [01:46<00:13, 4.48s/it2026-10-06 14:40:28,203 - INFO - Successfully downloaded: SP1.zip
Downloading files: 93%|▉| 26/28 [01:56<00:12, 6.18s/it2026-10-06 14:40:34,063 - INFO - Successfully downloaded: SP2_RM.zip
Downloading files: 96%|▉| 27/28 [02:02<00:06, 6.09s/it2026-10-06 14:40:37,100 - INFO - Successfully downloaded: TO.zip
Downloading files: 100%|█| 28/28 [02:05<00:00, 4.48s/it
2026-10-06 14:40:37,104 - INFO - Processing (depth 0): AC.zip
2026-10-06 14:40:37,223 - INFO - Extracted: b65dc3a4_AC_Amostra_Domicilios_12.txt
2026-10-06 14:40:37,225 - INFO - Extracted: b65dc3a4_AC_Amostra_Emigracao_12.txt
2026-10-06 14:40:37,227 - INFO - Extracted: b65dc3a4_AC_Amostra_Mortalidade_12.txt
2026-10-06 14:40:37,244 - INFO - Extracted: b65dc3a4_AC_Amostra_Pessoas_12.txt
2026-10-06 14:40:37,245 - INFO - Processing (depth 0): AL.zip
2026-10-06 14:40:37,612 - INFO - Extracted: 64f17b23_AL_Amostra_Domicilios_27.txt
2026-10-06 14:40:37,614 - INFO - Extracted: 64f17b23_AL_Amostra_Emigracao_27.txt
2026-10-06 14:40:37,615 - INFO - Extracted: 64f17b23_AL_Amostra_Mortalidade_27.txt
2026-10-06 14:40:37,666 - INFO - Extracted: 64f17b23_AL_Amostra_Pessoas_27.txt
2026-10-06 14:40:37,667 - INFO - Processing (depth 0): AM.zip
2026-10-06 14:40:37,949 - INFO - Extracted: 0e0ac84f_AM_Amostra_Domicilios_13.txt
2026-10-06 14:40:37,951 - INFO - Extracted: 0e0ac84f_AM_Amostra_Emigracao_13.txt
2026-10-06 14:40:37,952 - INFO - Extracted: 0e0ac84f_AM_Amostra_Mortalidade_13.txt
2026-10-06 14:40:37,997 - INFO - Extracted: 0e0ac84f_AM_Amostra_Pessoas_13.txt
2026-10-06 14:40:37,999 - INFO - Processing (depth 0): AP.zip
2026-10-06 14:40:38,080 - INFO - Extracted: 17c498d0_AP_Amostra_Domicilios_16.txt
2026-10-06 14:40:38,081 - INFO - Extracted: 17c498d0_AP_Amostra_Emigracao_16.txt
2026-10-06 14:40:38,083 - INFO - Extracted: 17c498d0_AP_Amostra_Mortalidade_16.txt
2026-10-06 14:40:38,096 - INFO - Extracted: 17c498d0_AP_Amostra_Pessoas_16.txt
2026-10-06 14:40:38,098 - INFO - Processing (depth 0): BA.zip
2026-10-06 14:40:39,572 - INFO - Extracted: 690b165a_BA_Amostra_Domicilios_29.txt
2026-10-06 14:40:39,574 - INFO - Extracted: 690b165a_BA_Amostra_Emigracao_29.txt
2026-10-06 14:40:39,576 - INFO - Extracted: 690b165a_BA_Amostra_Mortalidade_29.txt
2026-10-06 14:40:39,809 - INFO - Extracted: 690b165a_BA_Amostra_Pessoas_29.txt
2026-10-06 14:40:39,811 - INFO - Processing (depth 0): CE.zip
2026-10-06 14:40:40,614 - INFO - Extracted: 75e1f45b_CE_Amostra_Domicilios_23.txt
2026-10-06 14:40:40,616 - INFO - Extracted: 75e1f45b_CE_Amostra_Emigracao_23.txt
2026-10-06 14:40:40,617 - INFO - Extracted: 75e1f45b_CE_Amostra_Mortalidade_23.txt
2026-10-06 14:40:40,748 - INFO - Extracted: 75e1f45b_CE_Amostra_Pessoas_23.txt
2026-10-06 14:40:40,750 - INFO - Processing (depth 0): DF.zip
2026-10-06 14:40:40,879 - INFO - Extracted: a7bbca77_DF_Amostra_Domicilios_53.txt
2026-10-06 14:40:40,882 - INFO - Extracted: a7bbca77_DF_Amostra_Emigracao_53.txt
2026-10-06 14:40:40,883 - INFO - Extracted: a7bbca77_DF_Amostra_Mortalidade_53.txt
2026-10-06 14:40:40,902 - INFO - Extracted: a7bbca77_DF_Amostra_Pessoas_53.txt
2026-10-06 14:40:40,904 - INFO - Processing (depth 0): ES.zip
2026-10-06 14:40:41,304 - INFO - Extracted: 6eb433bd_ES_Amostra_Domicilios_32.txt
2026-10-06 14:40:41,306 - INFO - Extracted: 6eb433bd_ES_Amostra_Emigracao_32.txt
2026-10-06 14:40:41,307 - INFO - Extracted: 6eb433bd_ES_Amostra_Mortalidade_32.txt
2026-10-06 14:40:41,372 - INFO - Extracted: 6eb433bd_ES_Amostra_Pessoas_32.txt
2026-10-06 14:40:41,374 - INFO - Processing (depth 0): GO.zip
2026-10-06 14:40:42,075 - INFO - Extracted: fecb282b_GO_Amostra_Domicilios_52.txt
2026-10-06 14:40:42,077 - INFO - Extracted: fecb282b_GO_Amostra_Emigracao_52.txt
2026-10-06 14:40:42,079 - INFO - Extracted: fecb282b_GO_Amostra_Mortalidade_52.txt
2026-10-06 14:40:42,186 - INFO - Extracted: fecb282b_GO_Amostra_Pessoas_52.txt
2026-10-06 14:40:42,187 - INFO - Processing (depth 0): MA.zip
2026-10-06 14:40:42,943 - INFO - Extracted: e8ee0d2c_MA_Amostra_Domicilios_21.txt
2026-10-06 14:40:42,945 - INFO - Extracted: e8ee0d2c_MA_Amostra_Emigracao_21.txt
2026-10-06 14:40:42,947 - INFO - Extracted: e8ee0d2c_MA_Amostra_Mortalidade_21.txt
2026-10-06 14:40:43,067 - INFO - Extracted: e8ee0d2c_MA_Amostra_Pessoas_21.txt
2026-10-06 14:40:43,068 - INFO - Processing (depth 0): MG.zip
2026-10-06 14:40:45,484 - INFO - Extracted: 858e1e59_MG_Amostra_Domicilios_31.txt
2026-10-06 14:40:45,486 - INFO - Extracted: 858e1e59_MG_Amostra_Emigracao_31.txt
2026-10-06 14:40:45,488 - INFO - Extracted: 858e1e59_MG_Amostra_Mortalidade_31.txt
2026-10-06 14:40:45,873 - INFO - Extracted: 858e1e59_MG_Amostra_Pessoas_31.txt
2026-10-06 14:40:45,875 - INFO - Processing (depth 0): MS.zip
2026-10-06 14:40:46,173 - INFO - Extracted: 1658c49d_MS_Amostra_Domicilios_50.txt
2026-10-06 14:40:46,175 - INFO - Extracted: 1658c49d_MS_Amostra_Emigracao_50.txt
2026-10-06 14:40:46,176 - INFO - Extracted: 1658c49d_MS_Amostra_Mortalidade_50.txt
2026-10-06 14:40:46,217 - INFO - Extracted: 1658c49d_MS_Amostra_Pessoas_50.txt
2026-10-06 14:40:46,219 - INFO - Processing (depth 0): MT.zip
2026-10-06 14:40:46,602 - INFO - Extracted: 08aef50b_MT_Amostra_Domicilios_51.txt
2026-10-06 14:40:46,603 - INFO - Extracted: 08aef50b_MT_Amostra_Emigracao_51.txt
2026-10-06 14:40:46,605 - INFO - Extracted: 08aef50b_MT_Amostra_Mortalidade_51.txt
2026-10-06 14:40:46,660 - INFO - Extracted: 08aef50b_MT_Amostra_Pessoas_51.txt
2026-10-06 14:40:46,662 - INFO - Processing (depth 0): PA.zip
2026-10-06 14:40:47,346 - INFO - Extracted: 9019ff5b_PA_Amostra_Domicilios_15.txt
2026-10-06 14:40:47,348 - INFO - Extracted: 9019ff5b_PA_Amostra_Emigracao_15.txt
2026-10-06 14:40:47,350 - INFO - Extracted: 9019ff5b_PA_Amostra_Mortalidade_15.txt
2026-10-06 14:40:47,451 - INFO - Extracted: 9019ff5b_PA_Amostra_Pessoas_15.txt
2026-10-06 14:40:47,453 - INFO - Processing (depth 0): PB.zip
2026-10-06 14:40:48,045 - INFO - Extracted: bc0861e3_PB_Amostra_Domicilios_25.txt
2026-10-06 14:40:48,047 - INFO - Extracted: bc0861e3_PB_Amostra_Emigracao_25.txt
2026-10-06 14:40:48,049 - INFO - Extracted: bc0861e3_PB_Amostra_Mortalidade_25.txt
2026-10-06 14:40:48,143 - INFO - Extracted: bc0861e3_PB_Amostra_Pessoas_25.txt
2026-10-06 14:40:48,145 - INFO - Processing (depth 0): PE.zip
2026-10-06 14:40:49,012 - INFO - Extracted: a8df366c_PE_Amostra_Domicilios_26.txt
2026-10-06 14:40:49,014 - INFO - Extracted: a8df366c_PE_Amostra_Emigracao_26.txt
2026-10-06 14:40:49,015 - INFO - Extracted: a8df366c_PE_Amostra_Mortalidade_26.txt
2026-10-06 14:40:49,145 - INFO - Extracted: a8df366c_PE_Amostra_Pessoas_26.txt
2026-10-06 14:40:49,147 - INFO - Processing (depth 0): PI.zip
2026-10-06 14:40:49,618 - INFO - Extracted: a089c57e_PI_Amostra_Domicilios_22.txt
2026-10-06 14:40:49,621 - INFO - Extracted: a089c57e_PI_Amostra_Emigracao_22.txt
2026-10-06 14:40:49,622 - INFO - Extracted: a089c57e_PI_Amostra_Mortalidade_22.txt
2026-10-06 14:40:49,695 - INFO - Extracted: a089c57e_PI_Amostra_Pessoas_22.txt
2026-10-06 14:40:49,696 - INFO - Processing (depth 0): PR.zip
2026-10-06 14:40:50,957 - INFO - Extracted: 68798eb0_PR_Amostra_Domicilios_41.txt
2026-10-06 14:40:50,959 - INFO - Extracted: 68798eb0_PR_Amostra_Emigracao_41.txt
2026-10-06 14:40:50,960 - INFO - Extracted: 68798eb0_PR_Amostra_Mortalidade_41.txt
2026-10-06 14:40:51,135 - INFO - Extracted: 68798eb0_PR_Amostra_Pessoas_41.txt
2026-10-06 14:40:51,137 - INFO - Processing (depth 0): RJ.zip
2026-10-06 14:40:52,244 - INFO - Extracted: 64edc784_RJ_Amostra_Domicilios_33.txt
2026-10-06 14:40:52,246 - INFO - Extracted: 64edc784_RJ_Amostra_Emigracao_33.txt
2026-10-06 14:40:52,247 - INFO - Extracted: 64edc784_RJ_Amostra_Mortalidade_33.txt
2026-10-06 14:40:52,396 - INFO - Extracted: 64edc784_RJ_Amostra_Pessoas_33.txt
2026-10-06 14:40:52,398 - INFO - Processing (depth 0): RN.zip
2026-10-06 14:40:52,821 - INFO - Extracted: 6769ac16_RN_Amostra_Domicilios_24.txt
2026-10-06 14:40:52,823 - INFO - Extracted: 6769ac16_RN_Amostra_Emigracao_24.txt
2026-10-06 14:40:52,825 - INFO - Extracted: 6769ac16_RN_Amostra_Mortalidade_24.txt
2026-10-06 14:40:52,893 - INFO - Extracted: 6769ac16_RN_Amostra_Pessoas_24.txt
2026-10-06 14:40:52,894 - INFO - Processing (depth 0): RO.zip
2026-10-06 14:40:53,108 - INFO - Extracted: a9f49b63_RO_Amostra_Domicilios_11.txt
2026-10-06 14:40:53,110 - INFO - Extracted: a9f49b63_RO_Amostra_Emigracao_11.txt
2026-10-06 14:40:53,111 - INFO - Extracted: a9f49b63_RO_Amostra_Mortalidade_11.txt
2026-10-06 14:40:53,140 - INFO - Extracted: a9f49b63_RO_Amostra_Pessoas_11.txt
2026-10-06 14:40:53,142 - INFO - Processing (depth 0): RR.zip
2026-10-06 14:40:53,209 - INFO - Extracted: 29e3bef5_RR_Amostra_Domicilios_14.txt
2026-10-06 14:40:53,210 - INFO - Extracted: 29e3bef5_RR_Amostra_Emigracao_14.txt
2026-10-06 14:40:53,212 - INFO - Extracted: 29e3bef5_RR_Amostra_Mortalidade_14.txt
2026-10-06 14:40:53,223 - INFO - Extracted: 29e3bef5_RR_Amostra_Pessoas_14.txt
2026-10-06 14:40:53,225 - INFO - Processing (depth 0): RS.zip
2026-10-06 14:40:54,564 - INFO - Extracted: 711a9252_RS_Amostra_Domicilios_43.txt
2026-10-06 14:40:54,566 - INFO - Extracted: 711a9252_RS_Amostra_Emigracao_43.txt
2026-10-06 14:40:54,567 - INFO - Extracted: 711a9252_RS_Amostra_Mortalidade_43.txt
2026-10-06 14:40:54,758 - INFO - Extracted: 711a9252_RS_Amostra_Pessoas_43.txt
2026-10-06 14:40:54,759 - INFO - Processing (depth 0): SC.zip
2026-10-06 14:40:55,613 - INFO - Extracted: 3c1ce293_SC_Amostra_Domicilios_42.txt
2026-10-06 14:40:55,615 - INFO - Extracted: 3c1ce293_SC_Amostra_Emigracao_42.txt
2026-10-06 14:40:55,616 - INFO - Extracted: 3c1ce293_SC_Amostra_Mortalidade_42.txt
2026-10-06 14:40:55,727 - INFO - Extracted: 3c1ce293_SC_Amostra_Pessoas_42.txt
2026-10-06 14:40:55,730 - INFO - Processing (depth 0): SE.zip
2026-10-06 14:40:55,975 - INFO - Extracted: c7e763c0_SE_Amostra_Domicilios_28.txt
2026-10-06 14:40:55,977 - INFO - Extracted: c7e763c0_SE_Amostra_Emigracao_28.txt
2026-10-06 14:40:55,978 - INFO - Extracted: c7e763c0_SE_Amostra_Mortalidade_28.txt
2026-10-06 14:40:56,010 - INFO - Extracted: c7e763c0_SE_Amostra_Pessoas_28.txt
2026-10-06 14:40:56,012 - INFO - Processing (depth 0): SP1.zip
2026-10-06 14:40:58,342 - INFO - Extracted: 2fa85af7_SP1_Amostra_Domicilios_35_outras.txt
2026-10-06 14:40:58,344 - INFO - Extracted: 2fa85af7_SP1_Amostra_Emigracao_35_outras.txt
2026-10-06 14:40:58,346 - INFO - Extracted: 2fa85af7_SP1_Amostra_Mortalidade_35_outras.txt
2026-10-06 14:40:58,669 - INFO - Extracted: 2fa85af7_SP1_Amostra_Pessoas_35_outras.txt
2026-10-06 14:40:58,671 - INFO - Processing (depth 0): SP2_RM.zip
2026-10-06 14:40:59,834 - INFO - Extracted: cbe54e68_SP2-RM_Amostra_Domicilios_35_RMSP.txt
2026-10-06 14:40:59,836 - INFO - Extracted: cbe54e68_SP2-RM_Amostra_Emigracao_35_RMSP.txt
2026-10-06 14:40:59,838 - INFO - Extracted: cbe54e68_SP2-RM_Amostra_Mortalidade_35_RMSP.txt
2026-10-06 14:40:59,998 - INFO - Extracted: cbe54e68_SP2-RM_Amostra_Pessoas_35_RMSP.txt
2026-10-06 14:40:59,999 - INFO - Processing (depth 0): TO.zip
2026-10-06 14:41:00,262 - INFO - Extracted: e91b8629_TO_Amostra_Domicilios_17.txt
2026-10-06 14:41:00,264 - INFO - Extracted: e91b8629_TO_Amostra_Emigracao_17.txt
2026-10-06 14:41:00,266 - INFO - Extracted: e91b8629_TO_Amostra_Mortalidade_17.txt
2026-10-06 14:41:00,300 - INFO - Extracted: e91b8629_TO_Amostra_Pessoas_17.txt
Processing files: 100%|█| 112/112 [05:05<00:00, 2.72s/i
2026-10-06 14:46:05,335 - INFO - Successfully processed 112/112 files
2026-10-06 14:46:05,336 - INFO - Extraction completed successfully.
3. Harmonize the data#
Use the Harmonizer class from the socio4health library to harmonize the data. First, set the similarity threshold to 0.9, meaning that only variables with a similarity score of 0.9 or higher will be considered for harmonization. Next, use the s4h_vertical_merge method to merge the dataframes vertically.
har = Harmonizer()
har.similarity_threshold = 0.9
dfs = har.s4h_vertical_merge(bra_Censo_2010)
Grouping DataFrames: 100%|█| 112/112 [00:00<00:00, 137.4
Merging groups: 100%|█████| 1/1 [00:00<00:00, 1.88it/s]
After merging the dataframes, set the dictionary and the categories of interest. In this case, we are interested in the "Business" category. Then, use the s4h_data_selector method to filter the dataframes based on the dictionary, categories, and a key column (in this case 'V0001', which represents the state code). The s4h_data_selector method returns a list of filtered dataframes.
har.dict_df = dic
har.categories = ["Business"]
har.key_col = 'V0001'
filtered_ddfs = har.s4h_data_selector(dfs)
2026-10-06 14:46:06,702 - WARNING - key_col or key_val not defined, row-wise size will not be reduced
len(filtered_ddfs)
1
filtered_ddfs[0].compute()
| V0001 | V0208 | V0301 | V0222 | V0701 | V0211 | V0212 | M0201 | M0202 | M0203 | ... | M2011 | V0202 | V0221 | V0401 | V6531 | V6532 | V6530 | V6529 | V1005 | V2011 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 12 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | ... | <NA> | 1 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 201080 |
| 1 | 12 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | ... | <NA> | 8 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 200480 |
| 2 | 12 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | ... | <NA> | 1 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 200780 |
| 3 | 12 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | ... | <NA> | 8 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 200880 |
| 4 | 12 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | ... | <NA> | 1 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 201080 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 133867 | 17 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 3 | <NA> | ... | 1 | 1 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 042042 |
| 133868 | 17 | 0 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 3 | <NA> | ... | 2 | 3 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 046046 |
| 133869 | 17 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 3 | <NA> | ... | 1 | 1 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 037037 |
| 133870 | 17 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 1 | 0 | ... | 1 | 1 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 018018 |
| 133871 | 17 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 3 | <NA> | ... | 1 | 1 | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | <NA> | 041041 |
26993136 rows × 43 columns
Finally, we can perform some analysis on the harmonized data. In this case, we will calculate the total population by state (V0001) using the variable V0401, which represents the total population in each census tract. We will then create a horizontal bar plot to visualize the population distribution across states using matplotlib.
ddf = filtered_ddfs[0][["V0001", "V0401"]]
ddf = ddf.assign(
V0001 = ddf["V0001"].astype("category"),
V0401 = dd.to_numeric(ddf["V0401"], errors="coerce").astype("float64").fillna(0.0)
).categorize(columns=["V0001"])
pop = ddf.groupby("V0001")["V0401"].sum(split_out=8).compute()
pop = pop[pop>0]
row = dic.loc[dic["variable_name"]=="V0001", ["value","possible_answers"]].iloc[0]
vals = [s for s in re.split(r"\s*;\s*", str(row["value"]).strip(" ;")) if s]
labs = [s for s in re.split(r"\s*;\s*", str(row["possible_answers"]).strip(" ;")) if s]
idx = pop.index
if pd.api.types.is_integer_dtype(idx):
keys = [int(float(v)) for v in vals]
elif pd.api.types.is_float_dtype(idx):
keys = [float(v) for v in vals]
else:
keys = [str(int(float(v))) for v in vals]
if len(keys) != len(labs):
raise ValueError(f"Misalignment: {len(keys)} codes vs {len(labs)} names")
code2name = dict(zip(keys, labs))
pop_named = pop.rename(index=code2name)
top = pop_named.sort_values()
top_titled = top.copy()
top_titled.index = [str(s).title() for s in top.index]
fig, ax = plt.subplots(figsize=(11,7), dpi=130)
ax.barh(top_titled.index, top_titled.values)
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
ax.set_title(f"Population by State", pad=10)
ax.set_xlabel("Population")
ax.set_ylabel("State")
ax.xaxis.set_major_formatter(FuncFormatter(lambda x, p: f"{x/1e6:.1f} M"))
total = pop_named.sum()
for i, v in enumerate(top_titled.values):
ax.text(v, i, f"{v/1e6:.1f} M ({v/total:.1%})", va="center", ha="left", fontsize=9)
ax.grid(axis="x", linestyle="--", alpha=0.3)
plt.margins(x=0.03)
plt.tight_layout()
plt.show()
