[LangChain] Loader & Splitter

Loader

langchain_community 모듈에서 다양한 Document Loader 클래스를 이용해 데이터를 불러올 수 있음

from langchain_community.document_loaders import (TextLoader,
                                                  PyPDFLoader,
                                                  UnstructuredFileLoader)
    
# 1. TextLoader: 단순한 텍스트 파일(.txt) 로드
# - encoding 파라미터를 통해 텍스트 파일의 인코딩을 지정할 수 있음
# - utf-8 인코딩을 사용하면 한글, 유럽 언어 등 다양한 문자를 처리할 수 있음
text_loader = TextLoader('/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt', encoding='utf-8')

# load() 메서드를 호출하여 파일의 내용을 로드
# 결과는 Document 객체로 반환되며, 파일의 내용과 메타데이터를 포함
text_result = text_loader.load()


# 2. PyPDFLoader: PDF 파일 로드
# - PDF 문서를 파싱하여 텍스트 데이터를 추출
# - PDF 구조를 분석하여 문서를 효율적으로 처리
pdf_loader = PyPDFLoader('/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.pdf')

# load() 메서드를 호출하여 PDF 파일의 텍스트를 로드
pdf_result = pdf_loader.load()


# 3. UnstructuredFileLoader: 범용 로더
# - 파일 형식에 상관없이 다양한 종류의 데이터를 로드 가능
# - 지원되는 형식: 텍스트 파일(.txt), PDF, HTML, 이미지, 파워포인트 등
# - 내부적으로 파일의 형식을 감지하여 적절한 처리 방식을 자동으로 선택
ustrct_loader = UnstructuredFileLoader('/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt')

# load() 메서드를 호출하여 파일 내용을 로드
ustrct_result = ustrct_loader.load()
print(ustrct_result)

[Document(page_content='Chapter 1\n\nt was a bright cold day in April, and the clocks were strik- ing thirteen. Winston Smith, his chin nuzzled into his breast in an effort to escape the vile wind, slipped quickly through the glass doors of Victory Mansions, though not quickly enough to prevent a swirl of gritty dust from enter- ing along with him. The hallway smelt of boiled cabbage and old rag mats. At one end of it a coloured poster, too large for indoor display, had been tacked to the wall. It depicted simply an enor- mous face, more than a metre wide: the face of a man of about forty-five, with a heavy black moustache and rugged- ly handsome features. Winston made for the stairs. It was no use trying the lift. Even at the best of times it was sel- dom working, and at present the electric current was cut off during daylight hours. It was part of the economy drive in preparation for Hate Week. The flat was seven flights up, and Winston, who was thirty-nine and had a varicose ulcer above his right ankle, went slowly, resting several times on the way. On each landing, opposite the lift-shaft, the poster with the enormous face gazed from the wall. It was one of those pictures which are so contrived that the eyes follow you about when you move. BIG BROTHER IS WATCHING YOU, the caption beneath it ran. Inside the flat a fruity voice was reading out a list of fig-\n\nFree eBooks at Planet eBook.com\t3\n\nures which had something to do with the production of pig-iron. The voice came from an oblong metal plaque like a dulled mirror which formed part of the surface of the right-hand wall. Winston turned a switch and the voice sank somewhat, though the words were still distinguish- able. The instrument (the telescreen, it was called) could be dimmed, but there was no way of shutting it off complete- ly. He moved over to the window: a smallish, frail figure, the meagreness of his body merely emphasized by the blue overalls which were the uniform of the party. His hair was very fair, his face naturally sanguine, his skin roughened by coarse soap and blunt razor blades and the cold of the win- ter that had just ended. Outside, even through the shut window-pane, the world looked cold. Down in the street little eddies of wind were whirling dust and torn paper into spirals, and though the sun was shining and the sky a harsh blue, there seemed to be no colour in anything, except the posters that were plastered everywhere. The blackmoustachio’d face gazed down from every commanding corner. There was one on the house-front immediately opposite. BIG BROTHER IS WATCHING YOU, the caption said, while the dark eyes looked deep into Winston’s own. Down at street level an- other poster, torn at one corner, flapped fitfully in the wind, alternately covering and uncovering the single word IN- GSOC.\n\nIn the far distance a helicopter skimmed down between the roofs, hovered for an instant like a bluebottle, and darted away again with a curving flight. It was the po- lice patrol, snooping into people’s windows. The patrols did\n\n4\t1984\n\nnot matter, however. Only the Thought Police mattered. Behind Winston’s back the voice from the telescreen was still babbling away about pig-iron and the overfulfilment of the Ninth Three-Year Plan. The telescreen received and transmitted simultaneously. Any sound that Winston made, above the level of a very low whisper, would be picked up by it, moreover, so long as he remained within the field of vi- sion which the metal plaque commanded, he could be seen as well as heard. There was of course no way of knowing whether you were being watched at any given moment. How often, or on what system, the Thought Police plugged in on any individual wire was guesswork. It was even conceivable that they watched everybody all the time. But at any rate they could plug in your wire whenever they wanted to. You had to live—did live, from habit that became instinct—in the assumption that every sound you made was overheard, and, except in darkness, every movement scrutinized. Winston kept his back turned to the telescreen. It was safer, though, as he well knew, even a back can be revealing. A kilometre away the Ministry of Truth, his place of work, towered vast and white above the grimy landscape. This, he thought with a sort of vague distaste—this was London, chief city of Airstrip One, itself the third most populous of the provinces of Oceania. He tried to squeeze out some childhood memory that should tell him whether London had always been quite like this.\n\nWere there always these vis- tas of rotting nineteenth-century houses, their sides shored up with baulks of timber, their windows patched with card- board and their roofs with corrugated iron, their crazy\n\nFree eBooks at Planet eBook.com\t5\n\ngarden walls sagging in all directions? And the bombed sites where the plaster dust swirled in the air and the wil- low-herb straggled over the heaps of rubble; and the places where the bombs had cleared a larger patch and there had sprung up sordid colonies of wooden dwellings like chick- en-houses? But it was no use, he could not remember: nothing remained of his childhood except a series of bright- lit tableaux occurring against no background and mostly unintelligible. The Ministry of Truth—Minitrue, in Newspeak [New- speak was the official language of Oceania. For an account of its structure and etymology see Appendix. ]—was star- tlingly different from any other object in sight. It was an enormous pyramidal structure of glittering white con- crete, soaring up, terrace after terrace, 300 metres into the air. From where Winston stood it was just possible to read, picked out on its white face in elegant lettering, the three slogans of the Party:\n\nWAR IS PEACE FREEDOM IS SLAVERY IGNORANCE IS STRENGTH\n\nThe Ministry of Truth contained, it was said, three thousand rooms above ground level, and corresponding ramifications below. Scattered about London there were just three other buildings of similar appearance and size. So completely did they dwarf the surrounding architec- ture that from the roof of Victory Mansions you could see\n\n6\t1984\n\nall four of them simultaneously. They were the homes of the four Ministries between which the entire apparatus of government was divided. The Ministry of Truth, which concerned itself with news, entertainment, education, and the fine arts. The Ministry of Peace, which concerned itself with war. The Ministry of Love, which maintained law and order. And the Ministry of Plenty, which was responsible for economic affairs. Their names, in Newspeak: Minitrue, Minipax, Miniluv, and Miniplenty. The Ministry of Love was the really frightening one. There were no windows in it at all. Winston had never been inside the Ministry of Love, nor within half a kilometre of it. It was a place impossible to enter except on official business, and then only by penetrating through a maze of barbed- wire entanglements, steel doors, and hidden machine-gun nests. Even the streets leading up to its outer barriers were roamed by gorilla-faced guards in black uniforms, armed with jointed truncheons. Winston turned round abruptly. He had set his features into the expression of quiet optimism which it was advis- able to wear when facing the telescreen. He crossed the room into the tiny kitchen. By leaving the Ministry at this time of day he had sacrificed his lunch in the canteen, and he was aware that there was no food in the kitchen except a hunk of dark-coloured bread which had got to be saved for tomorrow’s breakfast.\n\nHe took down from the shelf a bottle of colourless liquid with a plain white label marked VICTORY GIN. It gave off a sickly, oily smell, as of Chinese rice-spirit. Winston poured out nearly a teacupful, nerved\n\nFree eBooks at Planet eBook.com\t7', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'})]

Splitter

문제	splitter 역할
AI 모델은 입력할 수 잇는 텍스트 크기에 제한이 있음	긴 문서를 모델이 처리 가능하도록 작은 '덩어리'로 나눔
텍스트를 나눌 때, 이전 내용과 연결이 끊기면 AI가 문맥을 이해하기 어려울 수 있음	덩어리마다 중복된 내용(chunk overlap)을 포함하여 문맥이 자연스럽게 이어지도록 도움
긴 텍스트를 처리하려면 시간이 오래 걸리고 메모리도 많이 소모됨	덩어리로 나눠 시간과 메모리 사용량을 줄임
텍스트가 단순히 줄바꿈으로만 나누는 게 아니라 특정 규칙에 따라 나뉘어야 할 때가 있음	텍스트를 특정 규칙이나 형식(문장, 단락, 토큰 등)으로 분할 할 수 있음

from langchain.text_splitter import (RecursiveCharacterTextSplitter, 
                                     CharacterTextSplitter)

# UnstructuredFileLoader를 사용하여 텍스트 파일을 로드
ustrct_loader = UnstructuredFileLoader('/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt')

"""
RecursiveCharacterTextSplitter
동작 방식:
- 텍스트를 분할할 때, 특정 기준(예: 공백, 문장 구분 기호 등)을 우선적으로 사용하여 가능한 자연스러운 단위로 텍스트를 나눔
- 설정된 기준으로 나눌 수 없는 경우, 더 작은 기준(예: 단어, 글자)으로 점진적으로 분할
- "큰 기준에서 작은 기준으로" 재귀적으로 탐색하며 텍스트를 분할하는 것이 특징

특징 및 장점:
- 문맥 유지: 가능한 한 문장을 자연스럽게 분할하며, 모델의 입력으로 적합한 덩어리를 만듦
- 유연성: 텍스트의 구조와 형식에 따라 다르게 작동하므로, 다양한 형식의 데이터에 적합함
- 효율성: 텍스트를 최대한 적은 조각으로 나누므로, 처리 속도와 메모리 사용이 상대적으로 효율적
"""

"""
from_tiktoken_encoder
- 텍스트 분할은 단순히 문자 수(len) 기준으로 분할하기보다는 토큰(token) 기준으로 분할하는 것이 효율적
- 토큰 단위로 분할을 수행하면, 사용 중인 언어 모델과 더 잘 연동될 수 있음 (ex. ChatGPT가 사용하는 토큰 단위와 유사한 기준으로 나뉘기 때문)
"""

# RecursiveCharacterTextSplitter 클래스를 사용하여 분할 객체를 생성
recur_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=200,    # 분할된 텍스트의 최대 길이(글자 수)를 200자로 설정
    chunk_overlap=50   # 분할 시, 앞 조각의 일부를 중복 포함시키는 크기를 50자로 설정
)

# 텍스트 로더에 위에서 설정한 분할기를 전달하여 텍스트를 분할
recur_split_result = ustrct_loader.load_and_split(text_splitter=recur_splitter)
print(recur_split_result)

"""
CharacterTextSplitter
동작 방식:
- 지정된 분리 기준(separator, 예: \n)을 기준으로 텍스트를 나눔
- 기준이 없는 경우, 텍스트를 단순히 chunk_size 길이로 잘라냄

특징 및 장점:
- 직관적: 설정된 기준만 따라 분할하므로 사용하기 쉽고, 특정 구조(예: 줄바꿈 기준)에 적합함
- 고정된 분할: 단순히 지정된 크기와 기준을 따르므로, 분할 결과를 예측하기 쉬움
- 빠른 처리: 별도의 재귀적 탐색 없이 설정된 기준만 따르기 때문에 속도가 빠름
"""
# CharacterTextSplitter 클래스를 사용하여 또 다른 분할 객체를 생성
char_splitter = CharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=200,       # 분할된 텍스트의 최대 길이(글자 수)를 200자로 설정
    chunk_overlap=100,    # 앞 조각의 중복 포함 크기를 100자로 설정
    separator='\n'        # 텍스트를 분할할 때 줄바꿈('\n') 기준으로 나누도록 설정
)

# 텍스트 로더에 새로운 분할기를 전달하여 텍스트를 분할
char_split_result =  ustrct_loader.load_and_split(text_splitter=char_splitter)
print(char_split_result)

[Document(page_content='Chapter 1', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='t was a bright cold day in April, and the clocks were strik- ing thirteen. Winston Smith, his chin nuzzled into his breast in an effort to escape the vile wind, slipped quickly through the glass doors of Victory Mansions, though not quickly enough to prevent a swirl of gritty dust from enter- ing along with him. The hallway smelt of boiled cabbage and old rag mats. At one end of it a coloured poster, too large for indoor display, had been tacked to the wall. It depicted simply an enor- mous face, more than a metre wide: the face of a man of about forty-five, with a heavy black moustache and rugged- ly handsome features. Winston made for the stairs. It was no use trying the lift. Even at the best of times it was sel- dom working, and at present the electric current was cut off during daylight hours. It was part of the economy drive in preparation for Hate', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='for the stairs. It was no use trying the lift. Even at the best of times it was sel- dom working, and at present the electric current was cut off during daylight hours. It was part of the economy drive in preparation for Hate Week. The flat was seven flights up, and Winston, who was thirty-nine and had a varicose ulcer above his right ankle, went slowly, resting several times on the way. On each landing, opposite the lift-shaft, the poster with the enormous face gazed from the wall. It was one of those pictures which are so contrived that the eyes follow you about when you move. BIG BROTHER IS WATCHING YOU, the caption beneath it ran. Inside the flat a fruity voice was reading out a list of fig-', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='Free eBooks at Planet eBook.com\t3', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='ures which had something to do with the production of pig-iron. The voice came from an oblong metal plaque like a dulled mirror which formed part of the surface of the right-hand wall. Winston turned a switch and the voice sank somewhat, though the words were still distinguish- able. The instrument (the telescreen, it was called) could be dimmed, but there was no way of shutting it off complete- ly. He moved over to the window: a smallish, frail figure, the meagreness of his body merely emphasized by the blue overalls which were the uniform of the party. His hair was very fair, his face naturally sanguine, his skin roughened by coarse soap and blunt razor blades and the cold of the win- ter that had just ended. Outside, even through the shut window-pane, the world looked cold. Down in the street little eddies of wind were whirling dust and torn paper into', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='and blunt razor blades and the cold of the win- ter that had just ended. Outside, even through the shut window-pane, the world looked cold. Down in the street little eddies of wind were whirling dust and torn paper into spirals, and though the sun was shining and the sky a harsh blue, there seemed to be no colour in anything, except the posters that were plastered everywhere. The blackmoustachio’d face gazed down from every commanding corner. There was one on the house-front immediately opposite. BIG BROTHER IS WATCHING YOU, the caption said, while the dark eyes looked deep into Winston’s own. Down at street level an- other poster, torn at one corner, flapped fitfully in the wind, alternately covering and uncovering the single word IN- GSOC.', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='In the far distance a helicopter skimmed down between the roofs, hovered for an instant like a bluebottle, and darted away again with a curving flight. It was the po- lice patrol, snooping into people’s windows. The patrols did\n\n4\t1984', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='not matter, however. Only the Thought Police mattered. Behind Winston’s back the voice from the telescreen was still babbling away about pig-iron and the overfulfilment of the Ninth Three-Year Plan. The telescreen received and transmitted simultaneously. Any sound that Winston made, above the level of a very low whisper, would be picked up by it, moreover, so long as he remained within the field of vi- sion which the metal plaque commanded, he could be seen as well as heard. There was of course no way of knowing whether you were being watched at any given moment. How often, or on what system, the Thought Police plugged in on any individual wire was guesswork. It was even conceivable that they watched everybody all the time. But at any rate they could plug in your wire whenever they wanted to. You had to live—did live, from habit that became instinct—in the assumption that every sound you made was overheard, and,', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='that they watched everybody all the time. But at any rate they could plug in your wire whenever they wanted to. You had to live—did live, from habit that became instinct—in the assumption that every sound you made was overheard, and, except in darkness, every movement scrutinized. Winston kept his back turned to the telescreen. It was safer, though, as he well knew, even a back can be revealing. A kilometre away the Ministry of Truth, his place of work, towered vast and white above the grimy landscape. This, he thought with a sort of vague distaste—this was London, chief city of Airstrip One, itself the third most populous of the provinces of Oceania. He tried to squeeze out some childhood memory that should tell him whether London had always been quite like this.', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='Were there always these vis- tas of rotting nineteenth-century houses, their sides shored up with baulks of timber, their windows patched with card- board and their roofs with corrugated iron, their crazy\n\nFree eBooks at Planet eBook.com\t5', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='garden walls sagging in all directions? And the bombed sites where the plaster dust swirled in the air and the wil- low-herb straggled over the heaps of rubble; and the places where the bombs had cleared a larger patch and there had sprung up sordid colonies of wooden dwellings like chick- en-houses? But it was no use, he could not remember: nothing remained of his childhood except a series of bright- lit tableaux occurring against no background and mostly unintelligible. The Ministry of Truth—Minitrue, in Newspeak [New- speak was the official language of Oceania. For an account of its structure and etymology see Appendix. ]—was star- tlingly different from any other object in sight. It was an enormous pyramidal structure of glittering white con- crete, soaring up, terrace after terrace, 300 metres into the air. From where Winston stood it was just possible to', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='different from any other object in sight. It was an enormous pyramidal structure of glittering white con- crete, soaring up, terrace after terrace, 300 metres into the air. From where Winston stood it was just possible to read, picked out on its white face in elegant lettering, the three slogans of the Party:', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='WAR IS PEACE FREEDOM IS SLAVERY IGNORANCE IS STRENGTH\n\nThe Ministry of Truth contained, it was said, three thousand rooms above ground level, and corresponding ramifications below. Scattered about London there were just three other buildings of similar appearance and size. So completely did they dwarf the surrounding architec- ture that from the roof of Victory Mansions you could see\n\n6\t1984', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='all four of them simultaneously. They were the homes of the four Ministries between which the entire apparatus of government was divided. The Ministry of Truth, which concerned itself with news, entertainment, education, and the fine arts. The Ministry of Peace, which concerned itself with war. The Ministry of Love, which maintained law and order. And the Ministry of Plenty, which was responsible for economic affairs. Their names, in Newspeak: Minitrue, Minipax, Miniluv, and Miniplenty. The Ministry of Love was the really frightening one. There were no windows in it at all. Winston had never been inside the Ministry of Love, nor within half a kilometre of it. It was a place impossible to enter except on official business, and then only by penetrating through a maze of barbed- wire entanglements, steel doors, and hidden machine-gun nests. Even the streets leading up to its outer barriers were roamed by gorilla-faced guards in', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='except on official business, and then only by penetrating through a maze of barbed- wire entanglements, steel doors, and hidden machine-gun nests. Even the streets leading up to its outer barriers were roamed by gorilla-faced guards in black uniforms, armed with jointed truncheons. Winston turned round abruptly. He had set his features into the expression of quiet optimism which it was advis- able to wear when facing the telescreen. He crossed the room into the tiny kitchen. By leaving the Ministry at this time of day he had sacrificed his lunch in the canteen, and he was aware that there was no food in the kitchen except a hunk of dark-coloured bread which had got to be saved for tomorrow’s breakfast.', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='He took down from the shelf a bottle of colourless liquid with a plain white label marked VICTORY GIN. It gave off a sickly, oily smell, as of Chinese rice-spirit. Winston poured out nearly a teacupful, nerved\n\nFree eBooks at Planet eBook.com\t7', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'})]
[Document(page_content='Chapter 1', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='t was a bright cold day in April, and the clocks were strik- ing thirteen. Winston Smith, his chin nuzzled into his breast in an effort to escape the vile wind, slipped quickly through the glass doors of Victory Mansions, though not quickly enough to prevent a swirl of gritty dust from enter- ing along with him. The hallway smelt of boiled cabbage and old rag mats. At one end of it a coloured poster, too large for indoor display, had been tacked to the wall. It depicted simply an enor- mous face, more than a metre wide: the face of a man of about forty-five, with a heavy black moustache and rugged- ly handsome features. Winston made for the stairs. It was no use trying the lift. Even at the best of times it was sel- dom working, and at present the electric current was cut off during daylight hours. It was part of the economy drive in preparation for Hate Week. The flat was seven flights up, and Winston, who was thirty-nine and had a varicose ulcer above his right ankle, went slowly, resting several times on the way. On each landing, opposite the lift-shaft, the poster with the enormous face gazed from the wall. It was one of those pictures which are so contrived that the eyes follow you about when you move. BIG BROTHER IS WATCHING YOU, the caption beneath it ran. Inside the flat a fruity voice was reading out a list of fig-', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='Free eBooks at Planet eBook.com\t3', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='ures which had something to do with the production of pig-iron. The voice came from an oblong metal plaque like a dulled mirror which formed part of the surface of the right-hand wall. Winston turned a switch and the voice sank somewhat, though the words were still distinguish- able. The instrument (the telescreen, it was called) could be dimmed, but there was no way of shutting it off complete- ly. He moved over to the window: a smallish, frail figure, the meagreness of his body merely emphasized by the blue overalls which were the uniform of the party. His hair was very fair, his face naturally sanguine, his skin roughened by coarse soap and blunt razor blades and the cold of the win- ter that had just ended. Outside, even through the shut window-pane, the world looked cold. Down in the street little eddies of wind were whirling dust and torn paper into spirals, and though the sun was shining and the sky a harsh blue, there seemed to be no colour in anything, except the posters that were plastered everywhere. The blackmoustachio’d face gazed down from every commanding corner. There was one on the house-front immediately opposite. BIG BROTHER IS WATCHING YOU, the caption said, while the dark eyes looked deep into Winston’s own. Down at street level an- other poster, torn at one corner, flapped fitfully in the wind, alternately covering and uncovering the single word IN- GSOC.', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='In the far distance a helicopter skimmed down between the roofs, hovered for an instant like a bluebottle, and darted away again with a curving flight. It was the po- lice patrol, snooping into people’s windows. The patrols did\n4\t1984', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='not matter, however. Only the Thought Police mattered. Behind Winston’s back the voice from the telescreen was still babbling away about pig-iron and the overfulfilment of the Ninth Three-Year Plan. The telescreen received and transmitted simultaneously. Any sound that Winston made, above the level of a very low whisper, would be picked up by it, moreover, so long as he remained within the field of vi- sion which the metal plaque commanded, he could be seen as well as heard. There was of course no way of knowing whether you were being watched at any given moment. How often, or on what system, the Thought Police plugged in on any individual wire was guesswork. It was even conceivable that they watched everybody all the time. But at any rate they could plug in your wire whenever they wanted to. You had to live—did live, from habit that became instinct—in the assumption that every sound you made was overheard, and, except in darkness, every movement scrutinized. Winston kept his back turned to the telescreen. It was safer, though, as he well knew, even a back can be revealing. A kilometre away the Ministry of Truth, his place of work, towered vast and white above the grimy landscape. This, he thought with a sort of vague distaste—this was London, chief city of Airstrip One, itself the third most populous of the provinces of Oceania. He tried to squeeze out some childhood memory that should tell him whether London had always been quite like this.', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='Were there always these vis- tas of rotting nineteenth-century houses, their sides shored up with baulks of timber, their windows patched with card- board and their roofs with corrugated iron, their crazy\nFree eBooks at Planet eBook.com\t5', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='garden walls sagging in all directions? And the bombed sites where the plaster dust swirled in the air and the wil- low-herb straggled over the heaps of rubble; and the places where the bombs had cleared a larger patch and there had sprung up sordid colonies of wooden dwellings like chick- en-houses? But it was no use, he could not remember: nothing remained of his childhood except a series of bright- lit tableaux occurring against no background and mostly unintelligible. The Ministry of Truth—Minitrue, in Newspeak [New- speak was the official language of Oceania. For an account of its structure and etymology see Appendix. ]—was star- tlingly different from any other object in sight. It was an enormous pyramidal structure of glittering white con- crete, soaring up, terrace after terrace, 300 metres into the air. From where Winston stood it was just possible to read, picked out on its white face in elegant lettering, the three slogans of the Party:', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='WAR IS PEACE FREEDOM IS SLAVERY IGNORANCE IS STRENGTH\nThe Ministry of Truth contained, it was said, three thousand rooms above ground level, and corresponding ramifications below. Scattered about London there were just three other buildings of similar appearance and size. So completely did they dwarf the surrounding architec- ture that from the roof of Victory Mansions you could see\n6\t1984', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='all four of them simultaneously. They were the homes of the four Ministries between which the entire apparatus of government was divided. The Ministry of Truth, which concerned itself with news, entertainment, education, and the fine arts. The Ministry of Peace, which concerned itself with war. The Ministry of Love, which maintained law and order. And the Ministry of Plenty, which was responsible for economic affairs. Their names, in Newspeak: Minitrue, Minipax, Miniluv, and Miniplenty. The Ministry of Love was the really frightening one. There were no windows in it at all. Winston had never been inside the Ministry of Love, nor within half a kilometre of it. It was a place impossible to enter except on official business, and then only by penetrating through a maze of barbed- wire entanglements, steel doors, and hidden machine-gun nests. Even the streets leading up to its outer barriers were roamed by gorilla-faced guards in black uniforms, armed with jointed truncheons. Winston turned round abruptly. He had set his features into the expression of quiet optimism which it was advis- able to wear when facing the telescreen. He crossed the room into the tiny kitchen. By leaving the Ministry at this time of day he had sacrificed his lunch in the canteen, and he was aware that there was no food in the kitchen except a hunk of dark-coloured bread which had got to be saved for tomorrow’s breakfast.', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'}), Document(page_content='He took down from the shelf a bottle of colourless liquid with a plain white label marked VICTORY GIN. It gave off a sickly, oily smell, as of Chinese rice-spirit. Winston poured out nearly a teacupful, nerved\nFree eBooks at Planet eBook.com\t7', metadata={'source': '/Users/sunbok/workspace/fullstack_gpt/files/1984_chapter_one.txt'})]