Harmonization of data#
Run the tutorial via free cloud platforms: 
This notebook provides you with a tutorial on how to process sociodemographic and economic data from online data sources from Brazil. This tutorial assumes you have an intermediate or advanced understanding of Python and data manipulation.
Setting up the enviornment#
To run this notebook, you need to have the following prerequisites:
Python 3.10+
Additionally, you need to install the socio4health and pandas package, which can be done using pip:
!pip install socio4health[all] ipywidgets -q
Import Libraries#
import pandas as pd
from socio4health import Extractor
from socio4health.enums.data_info_enum import BraColnamesEnum, BraColspecsEnum
from socio4health.harmonizer import Harmonizer
from socio4health.utils import harmonizer_utils
import tqdm as tqdm
Extracting data from Brazil#
In this example, we will extract the Brazilian National Continuous Household Sample Survey (PNADC) for the year 2024 from the Brazilian Institute of Geography and Statistics (IBGE) website.
bra_online_extractor = Extractor(input_path="https://ftp.ibge.gov.br/Trabalho_e_Rendimento/Pesquisa_Nacional_por_Amostra_de_Domicilios_continua/Trimestral/Microdados/2024/",
down_ext=['.txt','.zip'],
is_fwf=True,
colnames=BraColnamesEnum.PNADC.value,
colspecs=BraColspecsEnum.PNADC.value,
output_path="data",
depth=0)
Providing the raw dictionary#
We need to provide a raw dictionary to the harmonizer that contains the column names and their corresponding data types. This is necessary for the harmonization process, as it allows the harmonizer to understand the structure of the data. To know more about how to construct the raw dictionary, you can check the documentation.
raw_dict = pd.read_excel('raw_dictionary.xlsx')
The raw dictionary is then standardized using the s4h_standardize_dict method, which ensures that the dictionary is in a consistent format, making it easier to work with during the harmonization process.
dic = harmonizer_utils.s4h_standardize_dict(raw_dict)
F:\Users\EQUIPO\Documents\GitHub\socio4health\.venv\Lib\site-packages\socio4health\utils\harmonizer_utils.py:97: FutureWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
.apply(_process_group)\
Additionally, the content of columns of the dictionary can be translated into English using s4h_translate_column function from s4h_harmonizer_utils module. Translation is performed for facilitate the understanding and processing of the data.
⚠️
Warning:
s4h_translate_column method may take some time depending on the size of the dictionary and the number of columns to be translated. It is recommended to use this method only if you need the content of the columns in English for further processing or analysis.
dic = harmonizer_utils.s4h_translate_column(dic, "question", language="en")
dic = harmonizer_utils.s4h_translate_column(dic, "description", language="en")
dic = harmonizer_utils.s4h_translate_column(dic, "possible_answers", language="en")
2026-10-13 17:25:28,950 - INFO - HTTP Request: GET https://huggingface.co/api/agent-harnesses "HTTP/1.1 200 OK"
2026-10-13 17:25:29,044 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en "HTTP/1.1 200 OK"
2026-10-13 17:25:29,140 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:25:29,159 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/config.json?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Fconfig.json=&etag=%22d53a30938309af55bb3dc44e59034a3acfd620ba%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:29,176 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/config.json?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Fconfig.json=&etag=%22d53a30938309af55bb3dc44e59034a3acfd620ba%22 "HTTP/1.1 200 OK"
F:\Users\EQUIPO\Documents\GitHub\socio4health\.venv\Lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\EQUIPO\.cache\huggingface\hub\models--Helsinki-NLP--opus-mt-ROMANCE-en. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
warnings.warn(message)
2026-10-13 17:25:29,306 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-10-13 17:25:29,307 - WARNING - Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-10-13 17:25:29,326 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/tokenizer_config.json?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Ftokenizer_config.json=&etag=%2287cd76224ab570a3aadf39215a6344011b170e4e%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:29,346 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/tokenizer_config.json?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Ftokenizer_config.json=&etag=%2287cd76224ab570a3aadf39215a6344011b170e4e%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:29,465 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-10-13 17:25:29,564 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
2026-10-13 17:25:29,662 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/source.spm "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:25:29,680 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/source.spm?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Fsource.spm=&etag=%229080d95ae1341be221ff48919ea4a362970ac9a0%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:29,702 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/source.spm?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Fsource.spm=&etag=%229080d95ae1341be221ff48919ea4a362970ac9a0%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:29,855 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/target.spm "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:25:29,871 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/target.spm?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Ftarget.spm=&etag=%22090397efd72233e61e84abe704ff4d3146f451db%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:29,890 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/target.spm?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Ftarget.spm=&etag=%22090397efd72233e61e84abe704ff4d3146f451db%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:30,013 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/vocab.json "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:25:30,029 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/vocab.json?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Fvocab.json=&etag=%22da7a7e613f68cd0373521753d65267e3ae6c345f%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:30,048 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/vocab.json?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Fvocab.json=&etag=%22da7a7e613f68cd0373521753d65267e3ae6c345f%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:30,181 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/target_vocab.json "HTTP/1.1 404 Not Found"
2026-10-13 17:25:30,276 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/added_tokens.json "HTTP/1.1 404 Not Found"
2026-10-13 17:25:30,383 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/special_tokens_map.json "HTTP/1.1 404 Not Found"
2026-10-13 17:25:30,484 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/tokenizer.json "HTTP/1.1 404 Not Found"
2026-10-13 17:25:30,577 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/chat_template.jinja "HTTP/1.1 404 Not Found"
2026-10-13 17:25:30,993 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en "HTTP/1.1 200 OK"
2026-10-13 17:25:31,593 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/model.safetensors "HTTP/1.1 404 Not Found"
2026-10-13 17:25:31,685 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/model.safetensors.index.json "HTTP/1.1 404 Not Found"
2026-10-13 17:25:31,788 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/pytorch_model.bin "HTTP/1.1 302 Found"
F:\Users\EQUIPO\Documents\GitHub\socio4health\.venv\Lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\EQUIPO\.cache\huggingface\hub. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
warnings.warn(message)
2026-10-13 17:25:31,917 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/xet-read-token/e9ca9975e3972afd80732f08ce01d3a1339f47f8 "HTTP/1.1 200 OK"
2026-10-13 17:25:45,709 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/main/model.safetensors "HTTP/1.1 404 Not Found"
2026-10-13 17:25:45,819 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en "HTTP/1.1 200 OK"
2026-10-13 17:25:45,962 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/commits/main "HTTP/1.1 200 OK"
2026-10-13 17:25:46,075 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/discussions?p=0 "HTTP/1.1 200 OK"
2026-10-13 17:25:46,370 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/commits/refs%2Fpr%2F6 "HTTP/1.1 200 OK"
2026-10-13 17:25:46,636 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/refs%2Fpr%2F6/model.safetensors.index.json "HTTP/1.1 404 Not Found"
2026-10-13 17:25:46,674 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/e9ca9975e3972afd80732f08ce01d3a1339f47f8/generation_config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:25:46,700 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/generation_config.json?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Fgeneration_config.json=&etag=%22f07859fcab98712b231d115475990a962eebc851%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:46,716 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/Helsinki-NLP/opus-mt-ROMANCE-en/e9ca9975e3972afd80732f08ce01d3a1339f47f8/generation_config.json?%2FHelsinki-NLP%2Fopus-mt-ROMANCE-en%2Fresolve%2Fe9ca9975e3972afd80732f08ce01d3a1339f47f8%2Fgeneration_config.json=&etag=%22f07859fcab98712b231d115475990a962eebc851%22 "HTTP/1.1 200 OK"
2026-10-13 17:25:46,731 - INFO - HTTP Request: HEAD https://huggingface.co/Helsinki-NLP/opus-mt-ROMANCE-en/resolve/refs%2Fpr%2F6/model.safetensors "HTTP/1.1 302 Found"
2026-10-13 17:25:46,829 - INFO - HTTP Request: GET https://huggingface.co/api/models/Helsinki-NLP/opus-mt-ROMANCE-en/xet-read-token/ddfee805aaa57f4bd198f88e8832ba2b012f9ae2 "HTTP/1.1 200 OK"
question translated
description translated
possible_answers translated
dic
| variable_name | question | description | value | initial_position | final_position | size | possible_answers | question_en | description_en | possible_answers_en | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | S01017 | condição de ocupação do domicílio | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0; 7.0 | 412.0 | 412.0 | 1.0 | próprio de algum morador - já pago; próprio de... | condition of occupation of the home | NaN | own by some resident - already paid; own by a ... |
| 1 | V2010 | cor ou raça | NaN | 2.0; 4.0; 6.0; 8.0; 0.0; 9.0 | 106.0 | 106.0 | 1.0 | branca; preta; amarela; parda; indígena; sem d... | colour or race | NaN | white; black; yellow; brown; indigenous; no de... |
| 2 | V0607 | curso mais elevado que frequentou anteriormente | NaN | 2.0; 4.0; 5.0; 6.0; 7.0; 8.0; 9.0; 10.0; 11.0 | 110.0 | 111.0 | 2.0 | pré-escola; regular do ensino fundamental; edu... | higher course you have previously attended | NaN | pre-school; basic education; education of youn... |
| 3 | S01013 | destino do lixo domiciliar | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 400.0 | 400.0 | 1.0 | coletado diretamente por serviço de limpeza; c... | destination of household waste | NaN | collected directly by cleaning service; collec... |
| 4 | ESTRATO | estrato | uf (2) + número sequencial (6) + dv (1) | NaN | 12.0 | 20.0 | 9.0 | NaN | stratum | uf (2) + sequential number (6) + dv (1) | NaN |
| 5 | S01012A | forma de escoadouro do banheiro ou sanitário | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0; 7.0 | 399.0 | 399.0 | 1.0 | rede geral, rede pluvial; fossa séptica ligada... | form of draining of the bathroom or toilet | NaN | general network, rainfall; network-linked sept... |
| 6 | S01014 | forma de iluminação do domicílio | NaN | 1.0; 2.0 | 401.0 | 401.0 | 1.0 | utiliza ao menos uma fonte de energia eletrica... | home lighting form | NaN | uses at least one electric power source; does ... |
| 7 | V2009 | idade do morador na data de referência | NaN | NaN | 103.0 | 105.0 | 3.0 | NaN | age of the resident at the reference date | NaN | NaN |
| 8 | S01003 | material predominante na cobertura (telhado) d... | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0; 7.0 | 383.0 | 383.0 | 1.0 | telha; laje de concreto; madeira aparelhada; z... | predominant material on the roof cover of the ... | NaN | tile; concrete slab; equipped wood; zinc; wood... |
| 9 | S01002 | material predominante na construção das parede... | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 382.0 | 382.0 | 1.0 | alvenaria com revestimento/ taipa com revestim... | predominant material in the construction of th... | NaN | masonry with coating/taipa with coating; mason... |
| 10 | S01004 | material predominante no piso deste domicílio | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 384.0 | 384.0 | 1.0 | telha sem laje de concreto; telha com laje de ... | predominant material on the floor of this home | NaN | concrete slab-free tile; concrete slab tile; o... |
| 11 | CAPITAL | município da capital | NaN | 11.0; 12.0; 13.0; 14.0; 15.0; 16.0; 17.0; 21.0... | 5.0 | 6.0 | 2.0 | município de porto velho (ro); município de ri... | municipality of the capital | NaN | municipality of porto velho (ro); municipality... |
| 12 | V1032 | peso do domicílio | NaN | NaN | 58.0 | 72.0 | 15.0 | NaN | household weight | NaN | NaN |
| 13 | S01007 | proveniência da água canalizada utilizada no d... | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 389.0 | 389.0 | 1.0 | rede geral de distribuição; poço profundo ou a... | source of piped water used at home | NaN | general distribution network; deep or artisana... |
| 14 | V3001 | sabe ler e escrever | NaN | 1.0; 2.0 | 107.0 | 107.0 | 1.0 | sim; não | can read and write | NaN | Yes; no |
| 15 | V2007 | sexo | NaN | 1.0; 2.0 | 94.0 | 94.0 | 1.0 | homem; mulher | sex | NaN | male; female |
| 16 | V1022 | situação do domicílio | NaN | 1.0; 2.0 | 32.0 | 32.0 | 1.0 | urbana; rural | situation of the household | NaN | urban; rural |
| 17 | S01016B | tipo de combustível utilizado no fogão | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 411.0 | 411.0 | 1.0 | gás de botijão; gás canalizado; lenha; carvão;... | type of fuel used on the stove | NaN | bottling gas; piped gas; wood; coal; electrica... |
| 18 | S01001 | tipo do domicílio | NaN | 1.0; 2.0; 3.0 | 381.0 | 381.0 | 1.0 | casa; apartamento; habitação em casa de cômodo... | type of household | NaN | house; apartment; dwelling in rooms, corkscrew... |
The s4h_classify_rows method is then used to classify the rows of the standardized dictionary based on the content of the specified columns. This classification helps in organizing the data and making it easier to work with during the harmonization process. The MODEL_PATH parameter specifies the path to a pre-trained model that is used for classification. You can provide your own model or use the default one provided in the files folder. The model is a fine-tuned BERT model for text classification. You can find more details about the model in the documentation.
dic = harmonizer_utils.s4h_classify_rows(dic, "question_en", "description_en", "possible_answers_en",
new_column_name="category",
MODEL_PATH="dirreno/harmonize_bert_finetuned_classifier")
2026-10-13 17:27:11,774 - INFO - HTTP Request: GET https://huggingface.co/api/models/dirreno/harmonize_bert_finetuned_classifier "HTTP/1.1 200 OK"
2026-10-13 17:27:11,926 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:27:11,943 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/config.json?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Fconfig.json=&etag=%22ff3a809e5f16d320b9be172c32074edc22f22026%22 "HTTP/1.1 200 OK"
2026-10-13 17:27:11,962 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/config.json?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Fconfig.json=&etag=%22ff3a809e5f16d320b9be172c32074edc22f22026%22 "HTTP/1.1 200 OK"
F:\Users\EQUIPO\Documents\GitHub\socio4health\.venv\Lib\site-packages\huggingface_hub\file_download.py:149: UserWarning: `huggingface_hub` cache-system uses symlinks by default to efficiently store duplicated files but your machine does not support them in C:\Users\EQUIPO\.cache\huggingface\hub\models--dirreno--harmonize_bert_finetuned_classifier. Caching files will still work but in a degraded version that might require more space on your disk. This warning can be disabled by setting the `HF_HUB_DISABLE_SYMLINKS_WARNING` environment variable. For more details, see https://huggingface.co/docs/huggingface_hub/how-to-cache#limitations.
To support symlinks on Windows, you either need to activate Developer Mode or to run Python as an administrator. In order to activate developer mode, see this article: https://docs.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development
warnings.warn(message)
2026-10-13 17:27:12,107 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/model.safetensors "HTTP/1.1 302 Found"
2026-10-13 17:27:12,202 - INFO - HTTP Request: GET https://huggingface.co/api/models/dirreno/harmonize_bert_finetuned_classifier/xet-read-token/04a378f831c207100f9f6bf0f367a114c62d9c8a "HTTP/1.1 200 OK"
2026-10-13 17:27:31,201 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/tokenizer_config.json "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:27:31,225 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/tokenizer_config.json?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Ftokenizer_config.json=&etag=%223a1ddfa0b6cff03bfff84e8008766510272b075d%22 "HTTP/1.1 200 OK"
2026-10-13 17:27:31,245 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/tokenizer_config.json?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Ftokenizer_config.json=&etag=%223a1ddfa0b6cff03bfff84e8008766510272b075d%22 "HTTP/1.1 200 OK"
2026-10-13 17:27:31,379 - INFO - HTTP Request: GET https://huggingface.co/api/models/dirreno/harmonize_bert_finetuned_classifier/tree/main/additional_chat_templates?recursive=false&expand=false "HTTP/1.1 404 Not Found"
2026-10-13 17:27:31,499 - INFO - HTTP Request: GET https://huggingface.co/api/models/dirreno/harmonize_bert_finetuned_classifier/tree/main?recursive=true&expand=false "HTTP/1.1 200 OK"
2026-10-13 17:27:31,602 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/vocab.txt "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:27:31,616 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/vocab.txt?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Fvocab.txt=&etag=%22fb140275c155a9c7c5a3b3e0e77a9e839594a938%22 "HTTP/1.1 200 OK"
2026-10-13 17:27:31,633 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/vocab.txt?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Fvocab.txt=&etag=%22fb140275c155a9c7c5a3b3e0e77a9e839594a938%22 "HTTP/1.1 200 OK"
2026-10-13 17:27:31,761 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/tokenizer.json "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:27:31,780 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/tokenizer.json?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Ftokenizer.json=&etag=%22c17ed520ed8438736732a54957a69306b8822215%22 "HTTP/1.1 200 OK"
2026-10-13 17:27:31,797 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/tokenizer.json?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Ftokenizer.json=&etag=%22c17ed520ed8438736732a54957a69306b8822215%22 "HTTP/1.1 200 OK"
2026-10-13 17:27:32,016 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/added_tokens.json "HTTP/1.1 404 Not Found"
2026-10-13 17:27:32,114 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/special_tokens_map.json "HTTP/1.1 307 Temporary Redirect"
2026-10-13 17:27:32,136 - INFO - HTTP Request: HEAD https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/special_tokens_map.json?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Fspecial_tokens_map.json=&etag=%227520992f25914d962f0e2fd0e0566fc33d19ec59%22 "HTTP/1.1 200 OK"
2026-10-13 17:27:32,156 - INFO - HTTP Request: GET https://huggingface.co/api/resolve-cache/models/dirreno/harmonize_bert_finetuned_classifier/04a378f831c207100f9f6bf0f367a114c62d9c8a/special_tokens_map.json?%2Fdirreno%2Fharmonize_bert_finetuned_classifier%2Fresolve%2F04a378f831c207100f9f6bf0f367a114c62d9c8a%2Fspecial_tokens_map.json=&etag=%227520992f25914d962f0e2fd0e0566fc33d19ec59%22 "HTTP/1.1 200 OK"
2026-10-13 17:27:32,256 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/chat_template.jinja "HTTP/1.1 404 Not Found"
2026-10-13 17:27:32,374 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/processor_config.json "HTTP/1.1 404 Not Found"
2026-10-13 17:27:32,466 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/video_preprocessor_config.json "HTTP/1.1 404 Not Found"
2026-10-13 17:27:32,560 - INFO - HTTP Request: HEAD https://huggingface.co/dirreno/harmonize_bert_finetuned_classifier/resolve/04a378f831c207100f9f6bf0f367a114c62d9c8a/preprocessor_config.json "HTTP/1.1 404 Not Found"
dic
| variable_name | question | description | value | initial_position | final_position | size | possible_answers | question_en | description_en | possible_answers_en | category | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | S01017 | condição de ocupação do domicílio | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0; 7.0 | 412.0 | 412.0 | 1.0 | próprio de algum morador - já pago; próprio de... | condition of occupation of the home | NaN | own by some resident - already paid; own by a ... | Housing |
| 1 | V2010 | cor ou raça | NaN | 2.0; 4.0; 6.0; 8.0; 0.0; 9.0 | 106.0 | 106.0 | 1.0 | branca; preta; amarela; parda; indígena; sem d... | colour or race | NaN | white; black; yellow; brown; indigenous; no de... | Identification |
| 2 | V0607 | curso mais elevado que frequentou anteriormente | NaN | 2.0; 4.0; 5.0; 6.0; 7.0; 8.0; 9.0; 10.0; 11.0 | 110.0 | 111.0 | 2.0 | pré-escola; regular do ensino fundamental; edu... | higher course you have previously attended | NaN | pre-school; basic education; education of youn... | Education |
| 3 | S01013 | destino do lixo domiciliar | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 400.0 | 400.0 | 1.0 | coletado diretamente por serviço de limpeza; c... | destination of household waste | NaN | collected directly by cleaning service; collec... | Nonstandard job |
| 4 | ESTRATO | estrato | uf (2) + número sequencial (6) + dv (1) | NaN | 12.0 | 20.0 | 9.0 | NaN | stratum | uf (2) + sequential number (6) + dv (1) | NaN | Identification |
| 5 | S01012A | forma de escoadouro do banheiro ou sanitário | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0; 7.0 | 399.0 | 399.0 | 1.0 | rede geral, rede pluvial; fossa séptica ligada... | form of draining of the bathroom or toilet | NaN | general network, rainfall; network-linked sept... | Nonstandard job |
| 6 | S01014 | forma de iluminação do domicílio | NaN | 1.0; 2.0 | 401.0 | 401.0 | 1.0 | utiliza ao menos uma fonte de energia eletrica... | home lighting form | NaN | uses at least one electric power source; does ... | Business |
| 7 | V2009 | idade do morador na data de referência | NaN | NaN | 103.0 | 105.0 | 3.0 | NaN | age of the resident at the reference date | NaN | NaN | Housing |
| 8 | S01003 | material predominante na cobertura (telhado) d... | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0; 7.0 | 383.0 | 383.0 | 1.0 | telha; laje de concreto; madeira aparelhada; z... | predominant material on the roof cover of the ... | NaN | tile; concrete slab; equipped wood; zinc; wood... | Business |
| 9 | S01002 | material predominante na construção das parede... | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 382.0 | 382.0 | 1.0 | alvenaria com revestimento/ taipa com revestim... | predominant material in the construction of th... | NaN | masonry with coating/taipa with coating; mason... | Business |
| 10 | S01004 | material predominante no piso deste domicílio | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 384.0 | 384.0 | 1.0 | telha sem laje de concreto; telha com laje de ... | predominant material on the floor of this home | NaN | concrete slab-free tile; concrete slab tile; o... | Business |
| 11 | CAPITAL | município da capital | NaN | 11.0; 12.0; 13.0; 14.0; 15.0; 16.0; 17.0; 21.0... | 5.0 | 6.0 | 2.0 | município de porto velho (ro); município de ri... | municipality of the capital | NaN | municipality of porto velho (ro); municipality... | Identification |
| 12 | V1032 | peso do domicílio | NaN | NaN | 58.0 | 72.0 | 15.0 | NaN | household weight | NaN | NaN | Identification |
| 13 | S01007 | proveniência da água canalizada utilizada no d... | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 389.0 | 389.0 | 1.0 | rede geral de distribuição; poço profundo ou a... | source of piped water used at home | NaN | general distribution network; deep or artisana... | Nonstandard job |
| 14 | V3001 | sabe ler e escrever | NaN | 1.0; 2.0 | 107.0 | 107.0 | 1.0 | sim; não | can read and write | NaN | Yes; no | Nonstandard job |
| 15 | V2007 | sexo | NaN | 1.0; 2.0 | 94.0 | 94.0 | 1.0 | homem; mulher | sex | NaN | male; female | Identification |
| 16 | V1022 | situação do domicílio | NaN | 1.0; 2.0 | 32.0 | 32.0 | 1.0 | urbana; rural | situation of the household | NaN | urban; rural | Housing |
| 17 | S01016B | tipo de combustível utilizado no fogão | NaN | 1.0; 2.0; 3.0; 4.0; 5.0; 6.0 | 411.0 | 411.0 | 1.0 | gás de botijão; gás canalizado; lenha; carvão;... | type of fuel used on the stove | NaN | bottling gas; piped gas; wood; coal; electrica... | Business |
| 18 | S01001 | tipo do domicílio | NaN | 1.0; 2.0; 3.0 | 381.0 | 381.0 | 1.0 | casa; apartamento; habitação em casa de cômodo... | type of household | NaN | house; apartment; dwelling in rooms, corkscrew... | Housing |
Extracting the data#
The extract method of the Extractor class is used retrieve the data from the specified input path. It returns a list of dataframes, each dataframe corresponding to a file extracted from the path.
dfs = bra_online_extractor.s4h_extract()
2026-10-13 17:27:36,763 - INFO - ----------------------
2026-10-13 17:27:36,764 - INFO - Starting data extraction...
2026-10-13 17:27:36,764 - INFO - Extracting data in online mode...
2026-10-13 17:27:36,764 - INFO - Scraping URL: https://ftp.ibge.gov.br/Trabalho_e_Rendimento/Pesquisa_Nacional_por_Amostra_de_Domicilios_continua/Trimestral/Microdados/2024/ with depth 0
2026-10-13 17:27:39,952 - INFO - Spider completed successfully for URL: https://ftp.ibge.gov.br/Trabalho_e_Rendimento/Pesquisa_Nacional_por_Amostra_de_Domicilios_continua/Trimestral/Microdados/2024/
2026-10-13 17:27:39,959 - INFO - Downloading files to: data
Downloading files: 0%| | 0/4 [00:00<?, ?it/s]2026-10-13 17:27:53,981 - INFO - Successfully downloaded: PNADC_012024_20250815.zip
Downloading files: 25%|██▌ | 1/4 [00:14<00:42, 14.02s/it]2026-10-13 17:28:09,204 - INFO - Successfully downloaded: PNADC_022024_20260324.zip
Downloading files: 50%|█████ | 2/4 [00:29<00:29, 14.73s/it]2026-10-13 17:28:24,121 - INFO - Successfully downloaded: PNADC_032024_20250815.zip
Downloading files: 75%|███████▌ | 3/4 [00:44<00:14, 14.81s/it]2026-10-13 17:28:37,876 - INFO - Successfully downloaded: PNADC_042024_20250815.zip
Downloading files: 100%|██████████| 4/4 [00:57<00:00, 14.48s/it]
2026-10-13 17:28:37,878 - INFO - Processing (depth 0): PNADC_012024_20250815.zip
2026-10-13 17:28:56,575 - INFO - Extracted: 9cea2126_PNADC_012024.txt
2026-10-13 17:28:56,576 - INFO - Processing (depth 0): PNADC_022024_20260324.zip
2026-10-13 17:29:18,131 - INFO - Extracted: 6e9c9ee9_PNADC_022024.txt
2026-10-13 17:29:18,133 - INFO - Processing (depth 0): PNADC_032024_20250815.zip
2026-10-13 17:29:36,856 - INFO - Extracted: 341510e1_PNADC_032024.txt
2026-10-13 17:29:36,857 - INFO - Processing (depth 0): PNADC_042024_20250815.zip
2026-10-13 17:29:41,873 - INFO - Extracted: 79d04f0e_PNADC_042024.txt
Processing files: 100%|██████████| 4/4 [02:50<00:00, 42.72s/it]
2026-10-13 17:32:32,748 - INFO - Successfully processed 4/4 files
2026-10-13 17:32:32,749 - INFO - Extraction completed successfully.
Harmonizing the data#
First, we need to create an instance of the Harmonizer class.
har = Harmonizer()
After the dictionary is standardized and translated, it can be used to harmonize the data. For this, set the dict_df attribute of the Harmonizer instance to the standardized dictionary. This allows the harmonizer to use the information from the dictionary to process the dataframes.
har.dict_df = dic
Next, we can set the parameters for the harmonization process. The similarity_threshold parameter is used to set the threshold for the similarity among column names. The nan_threshold parameter is used to set the threshold for the number of NaN values allowed in a column. If a column has more NaN values than the specified threshold, it will be dropped from the final dataframe.
har.similarity_threshold = 0.9
har.nan_threshold = 1
The s4h_vertical_merge method merges dataframes vertically. This means the data frames will be concatenated along the rows and aligned if their column names meet the previously set similarity threshold. The available columns can be obtained using the s4h_get_available_columns method, which returns a list of column names present in all dataframes after vertical merging.
dfs = har.s4h_vertical_merge(dfs)
available_columns = har.s4h_get_available_columns(dfs)
available_columns
Grouping DataFrames: 100%|██████████| 4/4 [00:00<00:00, 15.87it/s]
Merging groups: 100%|██████████| 1/1 [00:00<00:00, 8.42it/s]
['ANO',
'CAPITAL',
'ESTRATO',
'FILENAME',
'POSEST',
'POSEST_SXI',
'RM_RIDE',
'TRIMESTRE',
'UF',
'UPA',
'V1008',
'V1014',
'V1016',
'V1022',
'V1023',
'V1027',
'V1028',
'V1028001',
'V1028002',
'V1028003',
'V1028004',
'V1028005',
'V1028006',
'V1028007',
'V1028008',
'V1028009',
'V1028010',
'V1028011',
'V1028012',
'V1028013',
'V1028014',
'V1028015',
'V1028016',
'V1028017',
'V1028018',
'V1028019',
'V1028020',
'V1028021',
'V1028022',
'V1028023',
'V1028024',
'V1028025',
'V1028026',
'V1028027',
'V1028028',
'V1028029',
'V1028030',
'V1028031',
'V1028032',
'V1028033',
'V1028034',
'V1028035',
'V1028036',
'V1028037',
'V1028038',
'V1028039',
'V1028040',
'V1028041',
'V1028042',
'V1028043',
'V1028044',
'V1028045',
'V1028046',
'V1028047',
'V1028048',
'V1028049',
'V1028050',
'V1028051',
'V1028052',
'V1028053',
'V1028054',
'V1028055',
'V1028056',
'V1028057',
'V1028058',
'V1028059',
'V1028060',
'V1028061',
'V1028062',
'V1028063',
'V1028064',
'V1028065',
'V1028066',
'V1028067',
'V1028068',
'V1028069',
'V1028070',
'V1028071',
'V1028072',
'V1028073',
'V1028074',
'V1028075',
'V1028076',
'V1028077',
'V1028078',
'V1028079',
'V1028080',
'V1028081',
'V1028082',
'V1028083',
'V1028084',
'V1028085',
'V1028086',
'V1028087',
'V1028088',
'V1028089',
'V1028090',
'V1028091',
'V1028092',
'V1028093',
'V1028094',
'V1028095',
'V1028096',
'V1028097',
'V1028098',
'V1028099',
'V1028100',
'V1028101',
'V1028102',
'V1028103',
'V1028104',
'V1028105',
'V1028106',
'V1028107',
'V1028108',
'V1028109',
'V1028110',
'V1028111',
'V1028112',
'V1028113',
'V1028114',
'V1028115',
'V1028116',
'V1028117',
'V1028118',
'V1028119',
'V1028120',
'V1028121',
'V1028122',
'V1028123',
'V1028124',
'V1028125',
'V1028126',
'V1028127',
'V1028128',
'V1028129',
'V1028130',
'V1028131',
'V1028132',
'V1028133',
'V1028134',
'V1028135',
'V1028136',
'V1028137',
'V1028138',
'V1028139',
'V1028140',
'V1028141',
'V1028142',
'V1028143',
'V1028144',
'V1028145',
'V1028146',
'V1028147',
'V1028148',
'V1028149',
'V1028150',
'V1028151',
'V1028152',
'V1028153',
'V1028154',
'V1028155',
'V1028156',
'V1028157',
'V1028158',
'V1028159',
'V1028160',
'V1028161',
'V1028162',
'V1028163',
'V1028164',
'V1028165',
'V1028166',
'V1028167',
'V1028168',
'V1028169',
'V1028170',
'V1028171',
'V1028172',
'V1028173',
'V1028174',
'V1028175',
'V1028176',
'V1028177',
'V1028178',
'V1028179',
'V1028180',
'V1028181',
'V1028182',
'V1028183',
'V1028184',
'V1028185',
'V1028186',
'V1028187',
'V1028188',
'V1028189',
'V1028190',
'V1028191',
'V1028192',
'V1028193',
'V1028194',
'V1028195',
'V1028196',
'V1028197',
'V1028198',
'V1028199',
'V1028200',
'V1029',
'V1033',
'V2001',
'V2003',
'V2005',
'V2007',
'V2008',
'V20081',
'V20082',
'V2009',
'V2010',
'V3001',
'V3002',
'V3002A',
'V3003',
'V3003A',
'V3004',
'V3005',
'V3005A',
'V3006',
'V3006A',
'V3007',
'V3008',
'V3009',
'V3009A',
'V3010',
'V3011',
'V3011A',
'V3012',
'V3013',
'V3013A',
'V3013B',
'V3014',
'V4001',
'V4002',
'V4003',
'V4004',
'V4005',
'V4006',
'V4006A',
'V4007',
'V4008',
'V40081',
'V40082',
'V40083',
'V4009',
'V4010',
'V4012',
'V40121',
'V4013',
'V40132',
'V40132A',
'V4014',
'V4015',
'V40151',
'V401511',
'V401512',
'V4016',
'V40161',
'V40162',
'V40163',
'V4017',
'V40171',
'V401711',
'V4018',
'V40181',
'V40182',
'V40183',
'V4019',
'V4020',
'V4021',
'V4022',
'V4024',
'V4025',
'V4026',
'V4027',
'V4028',
'V4029',
'V4032',
'V4033',
'V40331',
'V403311',
'V403312',
'V40332',
'V403321',
'V403322',
'V40333',
'V403331',
'V4034',
'V40341',
'V403411',
'V403412',
'V40342',
'V403421',
'V403422',
'V4039',
'V4039C',
'V4040',
'V40401',
'V40402',
'V40403',
'V4041',
'V4043',
'V40431',
'V4044',
'V4045',
'V4046',
'V4047',
'V4048',
'V4049',
'V4050',
'V40501',
'V405011',
'V405012',
'V40502',
'V405021',
'V405022',
'V40503',
'V405031',
'V4051',
'V40511',
'V405111',
'V405112',
'V40512',
'V405121',
'V405122',
'V4056',
'V4056C',
'V4057',
'V4058',
'V40581',
'V405811',
'V405812',
'V40582',
'V405821',
'V405822',
'V40583',
'V405831',
'V40584',
'V4059',
'V40591',
'V405911',
'V405912',
'V40592',
'V405921',
'V405922',
'V4062',
'V4062C',
'V4063',
'V4063A',
'V4064',
'V4064A',
'V4071',
'V4072',
'V4072A',
'V4073',
'V4074',
'V4074A',
'V4075A',
'V4075A1',
'V4076',
'V40761',
'V40762',
'V40763',
'V4077',
'V4078',
'V4078A',
'V4082',
'VD2002',
'VD2003',
'VD2004',
'VD2006',
'VD3004',
'VD3005',
'VD3006',
'VD4001',
'VD4002',
'VD4003',
'VD4004',
'VD4004A',
'VD4005',
'VD4007',
'VD4008',
'VD4009',
'VD4010',
'VD4011',
'VD4012',
'VD4013',
'VD4014',
'VD4015',
'VD4016',
'VD4017',
'VD4018',
'VD4019',
'VD4020',
'VD4023',
'VD4030',
'VD4031',
'VD4032',
'VD4033',
'VD4034',
'VD4035',
'VD4036',
'VD4037']
For the selection of rows from the data, we can use the s4h_data_selector method. To use this method we first need to assign the categories of our interest, which can either be one or a set of the following categories: Business, Educations, Fertility, Housing, Identification, Migration, Nonstandard job, Social Security .This method allows us to select specific rows from the data based on the values in a specified column. The key_col parameter specifies the column to be used for selection, and the key_val parameter specifies the values to be selected. In this case, we will select rows where the value in the UF column is equal to 14, which corresponds to the state of Pernambuco. This allows us to filter the data to include only records from this specific state.
har.categories = ["Business"]
har.key_col = 'UF'
har.key_val = ['14']
filtered_ddfs = har.s4h_data_selector(dfs)
2026-10-13 17:36:25,775 - WARNING - Only key column found in the filtered DataFrame. Use compare_with_dict() for more information.
Finally, we can join the filtered dataframes into a single dataframe using the s4h_join_data method. This method combines the data from the filtered dataframes into a single dataframe, aligning the columns based on their names. The resulting dataframe will contain all the columns that are present in the filtered dataframes, and it will be ready for further analysis or export as a CSV file.
joined_df = filtered_ddfs[0].compute()
available_cols = joined_df.columns.tolist()
print(f"Available columns: {available_cols}")
print(f"Shape of the DataFrame: {joined_df.shape}")
print(joined_df.head())
joined_df.to_csv("data/PNADC_2024_harmonized.csv", index=False)
Available columns: ['UF']
Shape of the DataFrame: (23590, 1)
UF
12881 14
12882 14
12883 14
12884 14
12885 14