[PintOS] Project 3 : Virtual Memory - Memory Mapped Files

chohk10·2023년 5월 23일
pintos운영체제정글
0

PintOS

목록 보기
6/8
post-custom-banner

4. Memory Mapped Files

In this section, you will implement memory-mapped pages. Unlike anonymous pages, memory-mapped pages are file-backed mappings. The contents in the page mirror data in some existing file. If a page fault occurs, a physical frame is immediately allocated and the contents are copied into the memory from the file. When memory-mapped pages are unmapped or swapped out, any change in the content is reflected in the file.

mmap and munmap System Call

Implement mmap and munmap, which are the two system calls for memory mapped files. Your VM system must load pages lazily in mmap regions and use the mmaped file itself as a backing store for the mapping. You should implement and use do_mmap and do_munmap defined in vm/file.c to implement these two system calls.

👉🏻 유저 프로세스는 mmap() 이라는 시스템콜 함수를 호출할 수 있다.
👉🏻 mmap()은 syscall number SYS_MMAP 을 가지고 syscall_handler에 도달하게 된다.
👉🏻 해당 syscall number에 대해서 mmap()을 호출해서 시스템콜 핸들링을 해줄 수 있도록 switch case를 추가해준다.
👉🏻 mmap()은 특정한 조건들을 걸러준 후 실제로 메모리 매핑을 할 수 있도록 do_mmap()을 호출해준다.

void *mmap (void *addr, size_t length, int writable, int fd, off_t offset);

Maps length bytes the file open as fd starting from offset byte into the process's virtual address space at addr. The entire file is mapped into consecutive virtual pages starting at addr. If the length of the file is not a multiple of PGSIZE, then some bytes in the final mapped page "stick out" beyond the end of the file. Set these bytes to zero when the page is faulted in, and discard them when the page is written back to disk. If successful, this function returns the virtual address where the file is mapped. On failure, it must return NULL which is not a valid address to map a file.

Function fails if :

  • file opened as fd has a length of zero bytes
  • addr is not page aligned
  • addr is already in use
  • addr is zero
  • length is zero
  • fd is stdin or stdout
void * do_mmap (void *addr, size_t length, int writable, 
                struct file *file, off_t offset);

Function fails if :

  • If the range of pages mapped overlaps any existing set of mapped pages, including the stack or pages mapped at executable load time. ==NOT SURE ABOUT THIS ONE...==

Things to implement :

  • If the length of the file is not multiple of PGSIZE, set the bytes sticking out in the page to zero when the page is faulted in. Discard these zero-fill when the page is written back to disk.
  • Memory-mapped pages should be also allocated in a lazy manner just like anonymous pages.
void munmap (void *addr);

Unmaps the mapping for the specified address range addr, which must be the virtual address returned by a previous call to mmap by the same process that has not yet been unmapped.

All mappings are implicitly unmapped when a process exits, whether via exit or by any other means.
프로세스가 죽으면 매핑되었던 파일들은 암시적으로(보이지 않지만) 자동으로 unmap이 된다.
👉🏻 process_exit()에서 supplemental_page_table_kill()을 호출하기 때문에 implicit 하게 unmap이 된다는 것 같다.
When a mapping is unmapped, whether implicitly or explicitly, all pages written to by the process are written back to the file, and pages not written must not be.
👉🏻 프로세스가 죽으면서 자동으로 unmap 되었든, 직접 unmap 시스템콜을 통해 unmap을 했든, 프로세스가 페이지에 쓴 것이 있다면, 디스크의 파일에도 그 내용이 반영되어야 한다. (write 한게 없는 페이지는 무시되어야 한다.)
The pages are then removed from the process's list of virtual pages.

Closing or removing a file does not unmap any of its mappings. Once created, a mapping is valid until munmap is called or the process exits, following the Unix convention. See Removing an Open File for more information. You should use the file_reopen function to obtain a separate and independent reference to the file for each of its mappings.

If two or more processes map the same file, there is no requirement that they see consistent data. Unix handles this by making the two mappings share the same physical page, and the mmap system call also has an argument allowing the client to specify whether the page is shared or private (i.e. copy-on-write).

You may want to modify vm_file_init and vm_file_initializer in vm/vm.c according to your needs.

테스트 결과

After implementing validation for mmap syscall

pass tests/vm/pt-grow-stack
pass tests/vm/pt-grow-bad
pass tests/vm/pt-big-stk-obj
pass tests/vm/pt-bad-addr
pass tests/vm/pt-bad-read
pass tests/vm/pt-write-code
FAIL tests/vm/pt-write-code2
pass tests/vm/pt-grow-stk-sc
pass tests/vm/page-linear
pass tests/vm/page-parallel
pass tests/vm/page-merge-seq
pass tests/vm/page-merge-par
pass tests/vm/page-merge-stk
FAIL tests/vm/page-merge-mm
pass tests/vm/page-shuffle
FAIL tests/vm/mmap-read
FAIL tests/vm/mmap-close
FAIL tests/vm/mmap-unmap
FAIL tests/vm/mmap-overlap
FAIL tests/vm/mmap-twice
FAIL tests/vm/mmap-write
FAIL tests/vm/mmap-ro
FAIL tests/vm/mmap-exit
FAIL tests/vm/mmap-shuffle
pass tests/vm/mmap-bad-fd
FAIL tests/vm/mmap-clean
FAIL tests/vm/mmap-inherit
pass tests/vm/mmap-misalign
pass tests/vm/mmap-null
pass tests/vm/mmap-over-code
pass tests/vm/mmap-over-data
pass tests/vm/mmap-over-stk
FAIL tests/vm/mmap-remove
pass tests/vm/mmap-zero
pass tests/vm/mmap-bad-fd2
pass tests/vm/mmap-bad-fd3
pass tests/vm/mmap-zero-len
FAIL tests/vm/mmap-off
pass tests/vm/mmap-bad-off
pass tests/vm/mmap-kernel
FAIL tests/vm/lazy-file
pass tests/vm/lazy-anon
FAIL tests/vm/swap-file
FAIL tests/vm/swap-anon
FAIL tests/vm/swap-iter
FAIL tests/vm/swap-fork
FAIL tests/vm/cow/cow-simple
21 of 141 tests failed.

mmap 구현 후 18개

pass tests/vm/pt-grow-stack
pass tests/vm/pt-grow-bad
pass tests/vm/pt-big-stk-obj
pass tests/vm/pt-bad-addr
pass tests/vm/pt-bad-read
pass tests/vm/pt-write-code
FAIL tests/vm/pt-write-code2
pass tests/vm/pt-grow-stk-sc
pass tests/vm/page-linear
pass tests/vm/page-parallel
pass tests/vm/page-merge-seq
pass tests/vm/page-merge-par
pass tests/vm/page-merge-stk
FAIL tests/vm/page-merge-mm
pass tests/vm/page-shuffle
FAIL tests/vm/mmap-read
FAIL tests/vm/mmap-close
FAIL tests/vm/mmap-unmap
pass tests/vm/mmap-overlap
FAIL tests/vm/mmap-twice
FAIL tests/vm/mmap-write
pass tests/vm/mmap-ro
FAIL tests/vm/mmap-exit
pass tests/vm/mmap-shuffle
pass tests/vm/mmap-bad-fd
FAIL tests/vm/mmap-clean
FAIL tests/vm/mmap-inherit
pass tests/vm/mmap-misalign
pass tests/vm/mmap-null
pass tests/vm/mmap-over-code
pass tests/vm/mmap-over-data
pass tests/vm/mmap-over-stk
FAIL tests/vm/mmap-remove
pass tests/vm/mmap-zero
pass tests/vm/mmap-bad-fd2
pass tests/vm/mmap-bad-fd3
pass tests/vm/mmap-zero-len
FAIL tests/vm/mmap-off
pass tests/vm/mmap-bad-off
FAIL tests/vm/mmap-kernel
FAIL tests/vm/lazy-file
pass tests/vm/lazy-anon
FAIL tests/vm/swap-file
FAIL tests/vm/swap-anon
FAIL tests/vm/swap-iter
pass tests/vm/swap-fork
FAIL tests/vm/cow/cow-simple
18 of 141 tests failed.

mmap-kernel 의 경우 validation이 잘못된 것 같아, validation 부분 수정

pass tests/vm/pt-grow-stack
pass tests/vm/pt-grow-bad
pass tests/vm/pt-big-stk-obj
pass tests/vm/pt-bad-addr
pass tests/vm/pt-bad-read
pass tests/vm/pt-write-code
FAIL tests/vm/pt-write-code2
pass tests/vm/pt-grow-stk-sc
pass tests/vm/page-linear
pass tests/vm/page-parallel
pass tests/vm/page-merge-seq
pass tests/vm/page-merge-par
pass tests/vm/page-merge-stk
FAIL tests/vm/page-merge-mm
pass tests/vm/page-shuffle
FAIL tests/vm/mmap-read
FAIL tests/vm/mmap-close
FAIL tests/vm/mmap-unmap
pass tests/vm/mmap-overlap
FAIL tests/vm/mmap-twice
FAIL tests/vm/mmap-write
pass tests/vm/mmap-ro
FAIL tests/vm/mmap-exit
pass tests/vm/mmap-shuffle
pass tests/vm/mmap-bad-fd
FAIL tests/vm/mmap-clean
FAIL tests/vm/mmap-inherit
pass tests/vm/mmap-misalign
pass tests/vm/mmap-null
pass tests/vm/mmap-over-code
pass tests/vm/mmap-over-data
pass tests/vm/mmap-over-stk
FAIL tests/vm/mmap-remove
pass tests/vm/mmap-zero
pass tests/vm/mmap-bad-fd2
pass tests/vm/mmap-bad-fd3
pass tests/vm/mmap-zero-len
FAIL tests/vm/mmap-off
pass tests/vm/mmap-bad-off
pass tests/vm/mmap-kernel
FAIL tests/vm/lazy-file
pass tests/vm/lazy-anon
FAIL tests/vm/swap-file
FAIL tests/vm/swap-anon
FAIL tests/vm/swap-iter
FAIL tests/vm/swap-fork
FAIL tests/vm/cow/cow-simple
18 of 141 tests failed.

kernel 주소 관련 validation은 해결.

팀원이 발견한 오류 : read/write syscall을 수행할 때 buffer로 사용되는 페이지 블록이 spt 테이블에 등록된 페이지이면 안된다. 따라서 이 부분을 확인해주는 코드 추가

pass tests/vm/pt-grow-stack
pass tests/vm/pt-grow-bad
pass tests/vm/pt-big-stk-obj
pass tests/vm/pt-bad-addr
pass tests/vm/pt-bad-read
pass tests/vm/pt-write-code
pass tests/vm/pt-write-code2
pass tests/vm/pt-grow-stk-sc
pass tests/vm/page-linear
pass tests/vm/page-parallel
pass tests/vm/page-merge-seq
pass tests/vm/page-merge-par
pass tests/vm/page-merge-stk
FAIL tests/vm/page-merge-mm
pass tests/vm/page-shuffle
FAIL tests/vm/mmap-read
FAIL tests/vm/mmap-close
FAIL tests/vm/mmap-unmap
pass tests/vm/mmap-overlap
FAIL tests/vm/mmap-twice
FAIL tests/vm/mmap-write
pass tests/vm/mmap-ro
FAIL tests/vm/mmap-exit
pass tests/vm/mmap-shuffle
pass tests/vm/mmap-bad-fd
FAIL tests/vm/mmap-clean
FAIL tests/vm/mmap-inherit
pass tests/vm/mmap-misalign
pass tests/vm/mmap-null
pass tests/vm/mmap-over-code
pass tests/vm/mmap-over-data
pass tests/vm/mmap-over-stk
FAIL tests/vm/mmap-remove
pass tests/vm/mmap-zero
pass tests/vm/mmap-bad-fd2
pass tests/vm/mmap-bad-fd3
pass tests/vm/mmap-zero-len
FAIL tests/vm/mmap-off
pass tests/vm/mmap-bad-off
pass tests/vm/mmap-kernel
FAIL tests/vm/lazy-file
pass tests/vm/lazy-anon
FAIL tests/vm/swap-file
FAIL tests/vm/swap-anon
FAIL tests/vm/swap-iter
FAIL tests/vm/swap-fork
FAIL tests/vm/cow/cow-simple
17 of 141 tests failed.

file_reopen을 for문을 돌면서 계속 하도록 잘못 만들었는데, file_reopen을 do_mmap 함수에서 한번만 할 수 있도록 수정

pass tests/vm/pt-grow-stack
pass tests/vm/pt-grow-bad
pass tests/vm/pt-big-stk-obj
pass tests/vm/pt-bad-addr
pass tests/vm/pt-bad-read
pass tests/vm/pt-write-code
pass tests/vm/pt-write-code2
pass tests/vm/pt-grow-stk-sc
pass tests/vm/page-linear
pass tests/vm/page-parallel
pass tests/vm/page-merge-seq
pass tests/vm/page-merge-par
pass tests/vm/page-merge-stk
FAIL tests/vm/page-merge-mm
pass tests/vm/page-shuffle
FAIL tests/vm/mmap-read
FAIL tests/vm/mmap-close
FAIL tests/vm/mmap-unmap
pass tests/vm/mmap-overlap
FAIL tests/vm/mmap-twice
FAIL tests/vm/mmap-write
pass tests/vm/mmap-ro
FAIL tests/vm/mmap-exit
pass tests/vm/mmap-shuffle
pass tests/vm/mmap-bad-fd
FAIL tests/vm/mmap-clean
FAIL tests/vm/mmap-inherit
pass tests/vm/mmap-misalign
pass tests/vm/mmap-null
pass tests/vm/mmap-over-code
pass tests/vm/mmap-over-data
pass tests/vm/mmap-over-stk
FAIL tests/vm/mmap-remove
pass tests/vm/mmap-zero
pass tests/vm/mmap-bad-fd2
pass tests/vm/mmap-bad-fd3
pass tests/vm/mmap-zero-len
FAIL tests/vm/mmap-off
pass tests/vm/mmap-bad-off
pass tests/vm/mmap-kernel
FAIL tests/vm/lazy-file
pass tests/vm/lazy-anon
FAIL tests/vm/swap-file
FAIL tests/vm/swap-anon
FAIL tests/vm/swap-iter
pass tests/vm/swap-fork
FAIL tests/vm/cow/cow-simple
16 of 141 tests failed.

이것도 마찬가지로 랜덤하게 맞는 애들이 운좋게 맞은 것 같음

file-backed initializer 에서 aux에 담아두었던 정보를 file-page로 가져올 수 있도록 구현
file-backed 페이지에 대한 destroy 구현
do-mmap에서 length 뿐만 아니라, file_length를 고려할 수 있도록 추가 👉🏻 가져오려는 length보다 file_length가 작거나, 보기 시작하려고 표시한 offset 보다 file_length가 작다면, 그만큼의 공간을 0으로 초기화해서 메모리에 map해두어야 한다고 한다. (근데 이러면 anonymous와 다른게 뭐지..? 기존의 파일 영역을 넘어서 메모리에 write을 하게 된다면 디스크에 저장되어있는 파일도 그만큼 크기를 늘려야 하는걸까?

pass tests/vm/pt-grow-stack
pass tests/vm/pt-grow-bad
pass tests/vm/pt-big-stk-obj
pass tests/vm/pt-bad-addr
pass tests/vm/pt-bad-read
pass tests/vm/pt-write-code
pass tests/vm/pt-write-code2
pass tests/vm/pt-grow-stk-sc
pass tests/vm/page-linear
pass tests/vm/page-parallel
pass tests/vm/page-merge-seq
pass tests/vm/page-merge-par
pass tests/vm/page-merge-stk
FAIL tests/vm/page-merge-mm
pass tests/vm/page-shuffle
pass tests/vm/mmap-read
pass tests/vm/mmap-close
FAIL tests/vm/mmap-unmap
pass tests/vm/mmap-overlap
pass tests/vm/mmap-twice
FAIL tests/vm/mmap-write
pass tests/vm/mmap-ro
FAIL tests/vm/mmap-exit
pass tests/vm/mmap-shuffle
pass tests/vm/mmap-bad-fd
pass tests/vm/mmap-clean
pass tests/vm/mmap-inherit
pass tests/vm/mmap-misalign
pass tests/vm/mmap-null
pass tests/vm/mmap-over-code
pass tests/vm/mmap-over-data
pass tests/vm/mmap-over-stk
pass tests/vm/mmap-remove
pass tests/vm/mmap-zero
pass tests/vm/mmap-bad-fd2
pass tests/vm/mmap-bad-fd3
pass tests/vm/mmap-zero-len
FAIL tests/vm/mmap-off
pass tests/vm/mmap-bad-off
pass tests/vm/mmap-kernel
pass tests/vm/lazy-file
pass tests/vm/lazy-anon
FAIL tests/vm/swap-file
FAIL tests/vm/swap-anon
FAIL tests/vm/swap-iter
FAIL tests/vm/swap-fork
FAIL tests/vm/cow/cow-simple
10 of 141 tests failed.

munmap 구현 을 추가했더니 16개 틀리는...ㅋㅋ
👉🏻 spt_remove_page - spt 테이블에서 페이지를 제거해주는 함수에서 해당 페이지의 spt_elem을 제거해주는 부분이 없어서 의문이었음 -> ASSERT 문 안에 hash_delete를 추가해주었음!! 이렇게 했더니 14개 틀리는걸로 줄어들었다

pass tests/vm/pt-grow-stack
pass tests/vm/pt-grow-bad
pass tests/vm/pt-big-stk-obj
pass tests/vm/pt-bad-addr
pass tests/vm/pt-bad-read
pass tests/vm/pt-write-code
pass tests/vm/pt-write-code2
pass tests/vm/pt-grow-stk-sc
pass tests/vm/page-linear
pass tests/vm/page-parallel
pass tests/vm/page-merge-seq
pass tests/vm/page-merge-par
pass tests/vm/page-merge-stk
FAIL tests/vm/page-merge-mm
pass tests/vm/page-shuffle
FAIL tests/vm/mmap-read
FAIL tests/vm/mmap-close
pass tests/vm/mmap-unmap
pass tests/vm/mmap-overlap
pass tests/vm/mmap-twice
FAIL tests/vm/mmap-write
pass tests/vm/mmap-ro
FAIL tests/vm/mmap-exit
pass tests/vm/mmap-shuffle
pass tests/vm/mmap-bad-fd
FAIL tests/vm/mmap-clean
pass tests/vm/mmap-inherit
pass tests/vm/mmap-misalign
pass tests/vm/mmap-null
pass tests/vm/mmap-over-code
pass tests/vm/mmap-over-data
pass tests/vm/mmap-over-stk
FAIL tests/vm/mmap-remove
pass tests/vm/mmap-zero
pass tests/vm/mmap-bad-fd2
pass tests/vm/mmap-bad-fd3
pass tests/vm/mmap-zero-len
FAIL tests/vm/mmap-off
pass tests/vm/mmap-bad-off
pass tests/vm/mmap-kernel
FAIL tests/vm/lazy-file
pass tests/vm/lazy-anon
FAIL tests/vm/swap-file
FAIL tests/vm/swap-anon
FAIL tests/vm/swap-iter
FAIL tests/vm/swap-fork
FAIL tests/vm/cow/cow-simple
14 of 141 tests failed.
/* Maps LENGTH of bytes the FILE from OFFSET byte into the process's
* virtual address space at ADDR. */
void *do_mmap (void *addr, size_t length, int writable, struct file *file, off_t offset) {
   struct file *new_file = file_reopen(file);
   
   off_t file_len = file_length(new_file);
   size_t read_bytes = file_len < offset ? 0 : file_len - offset;
   if (read_bytes > length)
      read_bytes = length;
   
   int page_cnt = DIV_ROUND_UP(length, PGSIZE);
   
   size_t zero_bytes = page_cnt * PGSIZE - read_bytes;
  
   void *page_addr = addr;
   int cnt = 0;
   for (cnt; cnt < page_cnt; cnt++) {
      size_t page_read_bytes = read_bytes > PGSIZE ? PGSIZE : read_bytes;
      size_t page_zero_bytes = PGSIZE - read_bytes;
        
      struct segment_info *aux = (struct segment_info *)malloc(sizeof(struct segment_info));
      if (!aux) {
         goto err;
      }
        
      aux->file = new_file;
      aux->ofs = offset;
      aux->read_bytes = page_read_bytes;
      aux->mmap_start_addr = addr;
        
      if (!vm_alloc_page_with_initializer(VM_FILE | VM_MMAP, page_addr, writable, lazy_load, aux)){
         free(aux);
         goto err;
      }
        
      read_bytes -= page_read_bytes;
      zero_bytes -= page_zero_bytes;
      page_addr += PGSIZE;
      offset += PGSIZE;
   }
     
   return addr;
err:
   for (int remove_cnt = 0; remove_cnt < cnt; remove_cnt++) {
      struct thread *curr = thread_current();
      spt_remove_page(&curr->spt, spt_find_page(&curr->spt, addr + remove_cnt * PGSIZE));
   }
   file_close(new_file);
   
   return NULL;
}

file_close의 경우 spt_remove_page에서 호출하는 vm_dealloc 함수에서도 해주고 있기 때문에 중복해서 호출하면 안될 줄 알았는데, file_close는 void 이기도 하고, file이 NULL이 아닌 경우에 대해서만 file close를 진행하기 때문에 여러번 불러도 문제가 없을 것 같다!
unmap에서도 혹시 모르니 file_close를 한번 더 불러주면 좋을 것 같다! 혹시 destroy 함수에서 놓치는 경우가 있을 수도 있으니까!

팀원이 작성한 코드 중에서 오타가 있는것을 겨우겨우 발견하고 고쳐서 돌려보았다.

void
spt_remove_page (struct supplemental_page_table *spt, struct page *page) {
ASSERT(hash_delete(&spt->spt_hash, &page->spt_elem) != NULL);
  
vm_dealloc_page (page);
return true;
}

hash_delete 가 NULL이 아닌걸 확인하는 부분에서 괄호가 잘못되어 있었다.
print를 하나하나 찍으면서 헤어나오지 못하는 곳을 찾아가다보면 문제가 되는 곳을 발견할 수 있고, 그제서야 오타나 틀린 문법이 보이는 것 같다.
역시 print가 최고의 디버깅 방법이 아닐까?

여기까지 디버깅하니 9개 남았다 !

pass tests/vm/pt-grow-stack
pass tests/vm/pt-grow-bad
pass tests/vm/pt-big-stk-obj
pass tests/vm/pt-bad-addr
pass tests/vm/pt-bad-read
pass tests/vm/pt-write-code
pass tests/vm/pt-write-code2
pass tests/vm/pt-grow-stk-sc
pass tests/vm/page-linear
pass tests/vm/page-parallel
pass tests/vm/page-merge-seq
FAIL tests/vm/page-merge-par
pass tests/vm/page-merge-stk
FAIL tests/vm/page-merge-mm
pass tests/vm/page-shuffle
pass tests/vm/mmap-read
pass tests/vm/mmap-close
FAIL tests/vm/mmap-unmap
pass tests/vm/mmap-overlap
pass tests/vm/mmap-twice
pass tests/vm/mmap-write
pass tests/vm/mmap-ro
FAIL tests/vm/mmap-exit
pass tests/vm/mmap-shuffle
pass tests/vm/mmap-bad-fd
pass tests/vm/mmap-clean
pass tests/vm/mmap-inherit
pass tests/vm/mmap-misalign
pass tests/vm/mmap-null
pass tests/vm/mmap-over-code
pass tests/vm/mmap-over-data
pass tests/vm/mmap-over-stk
pass tests/vm/mmap-remove
pass tests/vm/mmap-zero
pass tests/vm/mmap-bad-fd2
pass tests/vm/mmap-bad-fd3
pass tests/vm/mmap-zero-len
pass tests/vm/mmap-off
pass tests/vm/mmap-bad-off
pass tests/vm/mmap-kernel
pass tests/vm/lazy-file
pass tests/vm/lazy-anon
FAIL tests/vm/swap-file
FAIL tests/vm/swap-anon
FAIL tests/vm/swap-iter
FAIL tests/vm/swap-fork
FAIL tests/vm/cow/cow-simple
9 of 141 tests failed.

디버깅중 ㅋㅋㅋ

pintos -v -k -T 60 -m 20   --fs-disk=10 -p tests/vm/mmap-unmap:mmap-unmap -p ../../tests/vm/sample.txt:sample.txt --swap-disk=4 -- -q   -f run mmap-unmap
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
Kernel command line: -q -f put mmap-unmap put sample.txt run mmap-unmap
0 ~ 9fc00 1
100000 ~ 13e0000 1
Pintos booting with:
	base_mem: 0x0 ~ 0x9fc00 (Usable: 639 kB)
	ext_mem: 0x100000 ~ 0x13e0000 (Usable: 19,328 kB)
Calibrating timer...  71,987,200 loops/s.
hd0:0: detected 337 sector (168 kB) disk, model "QEMU HARDDISK", serial "QM00001"
hd0:1: detected 20,160 sector (9 MB) disk, model "QEMU HARDDISK", serial "QM00002"
hd1:0: detected 109 sector (54 kB) disk, model "QEMU HARDDISK", serial "QM00003"
hd1:1: detected 8,064 sector (3 MB) disk, model "QEMU HARDDISK", serial "QM00004"
Formatting file system...done.
Boot complete.
Putting 'mmap-unmap' into the file system...
Putting 'sample.txt' into the file system...
Executing 'mmap-unmap':
made a new page at : 0x400000
made a new page at : 0x401000
made a new page at : 0x402000
made a new page at : 0x403000
made a new page at : 0x404000
made a new page at : 0x604000
made a new page at : 0x605000
made a new page at : 0x4747f000
there was a page in spt table : 0x400d05
there was a page in spt table : 0x605608
there was a page in spt table : 0x4010fc
there was a page in spt table : 0x4044a0
there was a page in spt table : 0x402ef4
there was a page in spt table : 0x403000
(mmap-unmap) begin
(mmap-unmap) open "sample.txt"
(mmap-unmap) mmap "sample.txt"
mmap : 0x10000000
page_cnt : 2
made a new page at : 0x10000000
made a new page at : 0x10001000
there was a page in spt table : 0x10000000
(mmap-unmap) memory is readable 540884285
(mmap-unmap) memory is readable 540888381
unmap : 0x10000000
remove page at : 0x10000000
next page will be at : 0x10001000
is the next page in spt? : 1
finish unmapping
there was a page in spt table : 0x10001000
(mmap-unmap) unmapped memory is readable (0): FAILED
mmap-unmap: exit(1)
Execution of 'mmap-unmap' complete.
Timer: 72 ticks
Thread: 30 idle ticks, 30 kernel ticks, 12 user ticks
hd0:0: 0 reads, 0 writes
hd0:1: 114 reads, 256 writes
hd1:0: 109 reads, 0 writes
hd1:1: 0 reads, 0 writes
Console: 1994 characters output
Keyboard: 0 keys pressed
Exception: 0 page faults
Powering off...
pintos -v -k -T 60 -m 20   --fs-disk=10 -p tests/vm/mmap-unmap:mmap-unmap -p ../../tests/vm/sample.txt:sample.txt --swap-disk=4 -- -q   -f run mmap-unmap
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
Kernel command line: -q -f put mmap-unmap put sample.txt run mmap-unmap
0 ~ 9fc00 1
100000 ~ 13e0000 1
Pintos booting with:
	base_mem: 0x0 ~ 0x9fc00 (Usable: 639 kB)
	ext_mem: 0x100000 ~ 0x13e0000 (Usable: 19,328 kB)
Calibrating timer...  157,081,600 loops/s.
hd0:0: detected 337 sector (168 kB) disk, model "QEMU HARDDISK", serial "QM00001"
hd0:1: detected 20,160 sector (9 MB) disk, model "QEMU HARDDISK", serial "QM00002"
hd1:0: detected 109 sector (54 kB) disk, model "QEMU HARDDISK", serial "QM00003"
hd1:1: detected 8,064 sector (3 MB) disk, model "QEMU HARDDISK", serial "QM00004"
Formatting file system...done.
Boot complete.
Putting 'mmap-unmap' into the file system...
Putting 'sample.txt' into the file system...
Executing 'mmap-unmap':
made a new page at : 0x400000
made a new page at : 0x401000
made a new page at : 0x402000
made a new page at : 0x403000
made a new page at : 0x404000
made a new page at : 0x604000
made a new page at : 0x605000
made a new page at : 0x4747f000
there was a page in spt table : 0x400d05
there was a page in spt table : 0x605608
there was a page in spt table : 0x4010fc
there was a page in spt table : 0x4044a0
there was a page in spt table : 0x402ef4
there was a page in spt table : 0x403000
(mmap-unmap) begin
(mmap-unmap) open "sample.txt"
(mmap-unmap) mmap "sample.txt"
mmap : 0x10000000
page_cnt : 2
storing : 0x10000000
made a new page at : 0x10000000
storing : 0x10000000
made a new page at : 0x10001000
there was a page in spt table : 0x10000000
was the data transfer from aux to file_page successful? : 1
(mmap-unmap) memory is readable 540884285
(mmap-unmap) memory is readable 540888381
unmap : 0x10000000
starting address stored in the page : 0x10000000
remove page at : 0x10000000
next page will be at : 0x10001000
is the next page in spt? : 1
starting address stored in the page : 0x8004221f1b
break
is it because starting address is not the same? : 1
what is the starting address in the struct then? : 0x8004221f1b
finish unmapping
there was a page in spt table : 0x10001000
was the data transfer from aux to file_page successful? : 1
(mmap-unmap) unmapped memory is readable (0): FAILED
mmap-unmap: exit(1)
Execution of 'mmap-unmap' complete.
Timer: 80 ticks
Thread: 30 idle ticks, 32 kernel ticks, 18 user ticks
hd0:0: 0 reads, 0 writes
hd0:1: 114 reads, 256 writes
hd1:0: 109 reads, 0 writes
hd1:1: 0 reads, 0 writes
Console: 2379 characters output
Keyboard: 0 keys pressed
Exception: 0 page faults
Powering off...

After printing everything out, realized that the fault is occurring from an UNINIT page
팀원이 unmap을 한 후 page_fault가 나야하는데 나지 않아서 테스트가 실패한 것일테니까 page fault handler를 봐야하지 않겠냐 라고 이야기 해준 덕분에 디버깅하면서 흐름을 잡는데 도움이 되었다

디버깅을 위한 printf와 assert의 향연.....

/* Initialize the file backed page */
bool
file_backed_initializer (struct page *page, enum vm_type type, void *kva) {
	/* Set up the handler */
	page->operations = &file_ops;

	struct file_page *file_page = &page->file;

	struct segment_info *aux = page->uninit.aux;
	/* If aux is available, store the information. */
	if (aux != NULL) {
		file_page->file = aux->file;
		file_page->ofs = aux->ofs;
		file_page->read_bytes = aux->read_bytes;
		file_page->mmap_start_addr = aux->mmap_start_addr;
		printf("was the data transfer from aux to file_page successful? : %d\n", (file_page->mmap_start_addr == aux->mmap_start_addr));
	}

	/* Update page in-memory status. */
	page->in_memory = true;

	return true;
}

/* Swap in the page by read contents from the file. */
static bool
file_backed_swap_in (struct page *page, void *kva) {
	struct file_page *file_page UNUSED = &page->file;
}

/* Swap out the page by writeback contents to the file. */
static bool
file_backed_swap_out (struct page *page) {
	struct file_page *file_page UNUSED = &page->file;
}

/* Destory the file backed page. PAGE will be freed by the caller. */
static void
file_backed_destroy (struct page *page) {
	struct file_page *file_page = &page->file;

	/* If the page is already swapped out, don't write contents back again.
	   And there's no linked frame. */
	if (page->in_memory == false) {
		return;
	}

	/* If the page have been modified, write it back. */
	if (pml4_is_dirty(thread_current()->pml4, page->va)) {
		file_write_at(file_page->file, page->frame->kva, file_page->read_bytes, file_page->ofs);
		pml4_set_dirty(thread_current()->pml4, page->va, 0);
	}

	if (page->is_mmapped) {
		if (file_page->mmap_start_addr == page->va) {
			file_close(file_page->file);
		}
	} else {
		file_close(file_page->file);
	}
	page->in_memory = false;

	memset(page->frame->kva, 0, PGSIZE);
	pml4_clear_page(thread_current()->pml4, page->va);
	palloc_free_page(page->frame->kva);
	// ASSERT(!pml4e_walk(thread_current()->pml4, page->va, false));
	// ASSERT(!pml4e_walk(thread_current()->pml4, page->frame->kva, false));
	free(page->frame);
}

/* Maps LENGTH of bytes the FILE from OFFSET byte into the process's 
 * virtual address space at ADDR. */
void *do_mmap (void *addr, size_t length, int writable, struct file *file, off_t offset) {
	printf("mmap : %p\n", addr);
	struct file *new_file = file_reopen(file);

	off_t file_len = file_length(new_file);
	size_t read_bytes = file_len < offset ? 0 : file_len - offset;
	if (read_bytes > length)
		read_bytes = length;

	int page_cnt = DIV_ROUND_UP(length, PGSIZE);
	printf("page_cnt : %d\n", page_cnt);

	size_t zero_bytes = page_cnt * PGSIZE - read_bytes;

	void *page_addr = addr;
	int cnt = 0;
	for (cnt; cnt < page_cnt; cnt++) {
		size_t page_read_bytes = read_bytes > PGSIZE ? PGSIZE : read_bytes;
		size_t page_zero_bytes = PGSIZE - read_bytes;

		struct segment_info *aux = (struct segment_info *)malloc(sizeof(struct segment_info));
		if (!aux) {
			goto err;
		}

		// printf("hi\n");
		aux->file = new_file;
		aux->ofs = offset;
		aux->read_bytes = page_read_bytes;
		aux->mmap_start_addr = addr;
		printf("storing : %p\n", addr);

		if (!vm_alloc_page_with_initializer(VM_FILE | VM_MMAP, page_addr, writable, lazy_load, aux)){
			free(aux);
			goto err;
		}

		read_bytes -= page_read_bytes;
		zero_bytes -= page_zero_bytes;
		page_addr += PGSIZE;
		offset += PGSIZE;
	}

	return addr;
err:
	for (int remove_cnt = 0; remove_cnt < cnt; remove_cnt++) {
		struct thread *curr = thread_current();
		spt_remove_page(&curr->spt, spt_find_page(&curr->spt, addr + remove_cnt * PGSIZE));
	} 
	file_close(new_file); 

	return NULL;
}

/* Do the munmap */
void 
do_munmap (void *addr) {
	printf("unmap : %p\n", addr);
	void *page_addr = addr;
	struct page *page = spt_find_page(&thread_current()->spt, page_addr);
	ASSERT(page);
	ASSERT(page_get_type(page) == VM_FILE);
	ASSERT(page->file.mmap_start_addr == addr);

	while (page != NULL) {
		printf("starting address stored in the page : %p\n", page->file.mmap_start_addr);
		if (page_get_type(page) != VM_FILE || page->file.mmap_start_addr != addr) {	
			printf("break\n");
			printf("is it because starting address is not the same? : %d\n", (page->file.mmap_start_addr != addr));
			printf("what is the starting address in the struct then? : %p\n", page->file.mmap_start_addr);
			break;
		}

		ASSERT(spt_find_page(&thread_current()->spt, page_addr));
		printf("remove page at : %p\n", page_addr);
		spt_remove_page(&thread_current()->spt, page);
		ASSERT(!spt_find_page(&thread_current()->spt, page_addr));

		page_addr += PGSIZE;
		printf("next page will be at : %p\n", page_addr);
		page = spt_find_page(&thread_current()->spt, page_addr);
		printf("is the next page in spt? : %d\n", (page != NULL));
	}
	printf("finish unmapping\n");
}

UNINIT 페이지의 경우 aux에서 mmap_start_addr 값을 가져올 수 있도록 수정!!

void do_munmap (void *addr) {
	void *page_addr = addr;
	struct page *page = spt_find_page(&thread_current()->spt, page_addr);
	ASSERT(page);
	ASSERT(page_get_type(page) == VM_FILE);
	ASSERT(page->file.mmap_start_addr == addr);

	while (page != NULL && page->is_mmapped) {
		if (VM_TYPE(page->operations->type) == VM_UNINIT) {
			struct segment_info *aux = page->uninit.aux;
			if (aux->mmap_start_addr != addr) break;
		} else {
			if (page->file.mmap_start_addr != addr) break;
		}

		spt_remove_page(&thread_current()->spt, page);

		page_addr += PGSIZE;
		page = spt_find_page(&thread_current()->spt, page_addr);
	}
}

그 결과 하나를 해결해서 7~8개의 테스트 케이스가 남았다.
근데.. mmap-exit 케이스는 왜 해결되지 않은걸까 ㅠ 이번엔 이걸 파 보아야 할 것 같다 ㅜ 

pass tests/vm/pt-grow-stack
pass tests/vm/pt-grow-bad
pass tests/vm/pt-big-stk-obj
pass tests/vm/pt-bad-addr
pass tests/vm/pt-bad-read
pass tests/vm/pt-write-code
pass tests/vm/pt-write-code2
pass tests/vm/pt-grow-stk-sc
pass tests/vm/page-linear
pass tests/vm/page-parallel
pass tests/vm/page-merge-seq
pass tests/vm/page-merge-par
pass tests/vm/page-merge-stk
FAIL tests/vm/page-merge-mm
pass tests/vm/page-shuffle
pass tests/vm/mmap-read
pass tests/vm/mmap-close
pass tests/vm/mmap-unmap
pass tests/vm/mmap-overlap
pass tests/vm/mmap-twice
pass tests/vm/mmap-write
pass tests/vm/mmap-ro
FAIL tests/vm/mmap-exit
pass tests/vm/mmap-shuffle
pass tests/vm/mmap-bad-fd
pass tests/vm/mmap-clean
pass tests/vm/mmap-inherit
pass tests/vm/mmap-misalign
pass tests/vm/mmap-null
pass tests/vm/mmap-over-code
pass tests/vm/mmap-over-data
pass tests/vm/mmap-over-stk
pass tests/vm/mmap-remove
pass tests/vm/mmap-zero
pass tests/vm/mmap-bad-fd2
pass tests/vm/mmap-bad-fd3
pass tests/vm/mmap-zero-len
pass tests/vm/mmap-off
pass tests/vm/mmap-bad-off
pass tests/vm/mmap-kernel
pass tests/vm/lazy-file
pass tests/vm/lazy-anon
FAIL tests/vm/swap-file
FAIL tests/vm/swap-anon
FAIL tests/vm/swap-iter
FAIL tests/vm/swap-fork
FAIL tests/vm/cow/cow-simple
7 of 141 tests failed.

mmap-exit 테스트 케이스를 위해 print를 여려군데 찍어본 결과 process_exit 함수에서 child 프로세스를 죽일 때 proxess_cleanup까지 가지 못해서 spt 테이블을 죽이고, 변경된 mmap 페이지에 대해서 디스크에 write하는 과정을 거치지 않아서 테스트가 실패하는 것으로 확인하였다.
supplemental_page_table_kill 등을 수행하는 page_cleanup을 들어가기 전에 fd들을 close 하고 exit_sema, wait_sema 등을 조작하고 있는 현재 코드에서 (해당 코드는 팀원 한명이 지난 프로젝트 때 다른 팀원들과 작성했던 것이었다.) page_cleanup을 진행한 후에 fd와 sema를 조작해줄 수 있도록 수정해주었다.
그랬더니 이번에는 threads 관련 테스트 케이스들이 대거 fail 하였다. 

Kernel PANIC at ../../lib/kernel/list.c:142 in list_head(): assertion `list != NULL' failed.
Call stack: 0x8004217f56 0x80042183c0 0x800421a99b 0x8004221c7b 0x800421bfd2 0x800421bcd9 0x8004207278 0x8004207662.
0x0000008004217f56: debug_panic (lib/kernel/debug.c:32)
0x00000080042183c0: list_head (lib/kernel/list.c:143)
0x000000800421a99b: hash_first (lib/kernel/hash.c:191)
0x0000008004221c7b: supplemental_page_table_kill (vm/vm.c:341)
0x000000800421bfd2: process_cleanup (userprog/process.c:443)
0x000000800421bcd9: process_exit (userprog/process.c:358)
0x0000008004207278: thread_exit (threads/thread.c:339)
0x0000008004207662: init_thread (threads/thread.c:498)

supplemental_page_table_kill 이 현재는 hash iterator를 활용해서 구현되어있는데.. hash_clear 또는 hash_destroy로 바꿔서 구현해보면 좋을 것 같다 생각하고 있었다.
hash_first를 직접 사용하는 경우 위 assertion 메세지와 같이 bucket list가 비어있는지 하나하나 확인이 어렵다.
hash_destroy로 수정을 해도 위와 같이 list_head가 비어있는 오류가 발생했었다.
👉🏻 hash_destroy를 사용하는 경우, hash table 자체를 없애주기 때문에 spt_kill을 호출하는 process_cleanup을 proxess_exec에서 호출하는 경우, 테이블을 삭제해준 후 다시 spt 테이블을 init 해주어야 한다.
👉🏻 process_cleanup이 process_exit에서도, process_exec에서도 사용되기 때문에 이렇게 설계를 해주게 되었다. process_exit, 즉 프로세스를 종료할 때에, spt 테이블 구조체를 남겨주고 종료하면 메모리 누수가 발생할 수 있기 때문에 완전하게 테이블 구조체인 해시 구조체를 삭제해주는 것이 맞다고 생각했다. process_fork를 통해 spt_copy를 호출하게 되면 부모의 spt 테이블이 그대로 자식 프로세스에서도 생성되게 되는데, 자식이 fork한 그대로 사용하기 않고 process_exec를 하게 된다면 복제해온 spt 테이블의 내용을 비우고 사용할 수 있어야 한다. 이때 process_cleanup에서 spt 자체를 삭제하도록 앞서 구현을 했어야 했기에 spt_init을 한번 더 호출해줌으로서 proces_exec를 한 자식 프로세스가 정상 작동할 수 있다.

profile
chohk10
이전 포스트

[PintOS] Project 3 : Virtual Memory - Stack Growth

다음 포스트

[PintOS] Project 3 : Virtual Memory - Swap In/Out

post-custom-banner

0개의 댓글