Shared Memory

mmap

系统调用 mmap() 用处：

进程创建匿名的内存映射，把内存的物理页映射到进程的虚拟地址空间
进程把文件映射到进程的虚拟地址空间，可以向访问内存一样访问文件，不需要调用系统调用 read 和 write 访问文件，从而避免用户模式和内核模式之间的切换，提高读写文件的速度。
两个进程针对同一个i文件创建共享的内存映射，实现共享内存。

MMAP is a UNIX system call that maps files into memory. It’s a method used for memory-mapped file I/O. It brings in the optimization of lazy loading or demand paging such that the I/O or reading file doesn’t happen when the memory allocation is done, but when the memory is accessed. After the memory is no longer needed it can be cleared with munmap call. MMAP supports certain flags or argument which makes it suitable for allocating memory without file mapping as well. In Linux kernel, the malloc call uses mmap with MAP_ANONYMOUS flag for large allocations.

mmap definition

1

void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);

The addr specifies the starting address of the allocation and if it’s passed as NULL the kernel chooses the starting address.
The length argument specifies the length of allocation in bytes and should be > 0.
The prot argument describes the protection level
- PROT_EXEC: Pages may be executed.
- PROT_READ: Pages may be read.
- PROT_WRITE: Pages may be written.
- PROT_NONE: Pages may be not be accessed.
The fd indicates the file descriptor that represents the file backing the memory. This is not needed when using MAP_ANON flag and can be set to -1.
The offset indicates the offset in bytes from which the file should be read to memory. The allocated memory will load offset to offset + length bytes from the file when the memory access happens.

The flags argument is used to do various customizations, the interesting ones are:

MAP_SHARED or MAP_PRIVATE: determines if the pages would be shared across processes or owned by a single process.
MAP_ANONYMOUS (or MAP_ANON): is used to indicate that the pages are not backed by any file and are anonymous. malloc could use this flag to create a large memory allocation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


    // This is not a real implementation but meant for giving an idea
    void *malloc(size_t length) {
        return mmap(
            /* addr= */ NULL,
            /* length= */ length,
            /* prot= */ PROT_EXEC | PROT_READ | PROT_WRITE,
            /* flags= */ MAP_ANONYMOUS,
            /* fd */ -1, // Passing an invalid file descriptor
            /* offset= */ 0);
    }

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


    // Creating a file backed memory allocation
    struct stat statbuf;
    int fd = open("/some/shared/resource/file", "r");
    fstat(fd, &statbuf);

    // Creating a read only, shared, file backed memory allocation
    int *shared = mmap(
        /* addr= */ NULL,
        /* length= */ statbuf.st_size - 1,
        /* prot= */ PROT_READ,
        /* flags= */ MAP_SHARED,
        /* fd= */ fd,
        /* offset= */ 0);

munmap definition

1

int munmap(void *addr, size_t length);

The addr is the address of allocation to free, essentially what you got from calling the mmap(). After calling munmap(), any access on the memory address shall raise SIGSEV errors.
The length determines the area of memory to clear. The area of memory from addr to addr + length would be freed on this call.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


int main(void) {
    struct stat statbuf;
    int fp = open("/some/shared/resource/file", "r");
    fstat(fp, &statbuf);

    int *shared = mmap(
        /* addr= */ NULL,
        /* length= */ statbuf.st_size - 1,
        /* prot= */ PROT_READ | PROT_WRITE,
        /* flags= */ MAP_SHARED,
        /* fd= */ fp,
        /* offset= */ 0);

    pid_t child;
    if ((child = fork()) == 0) {
        // process forked. The child can access the 'shared'
        // object and perform read and write operations on it.
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <string.h>
#include <unistd.h>

void* create_shared_memory(size_t size) {
  // Our memory buffer will be readable and writable:
  int protection = PROT_READ | PROT_WRITE;

  // The buffer will be shared (meaning other processes can access it), but
  // anonymous (meaning third-party processes cannot obtain an address for it),
  // so only this process and its children will be able to use it:
  int visibility = MAP_SHARED | MAP_ANONYMOUS;

  // The remaining parameters to `mmap()` are not important for this use case,
  // but the manpage for `mmap` explains their purpose.
  return mmap(NULL, size, protection, visibility, -1, 0);
}

int main() {
  char parent_message[] = "hello";  // parent process will write this message
  char child_message[] = "goodbye"; // child process will then write this one

  void* shmem = create_shared_memory(128);

  memcpy(shmem, parent_message, sizeof(parent_message));

  int pid = fork();

  if (pid == 0) {
    printf("Child read: %s\n", shmem);
    memcpy(shmem, child_message, sizeof(child_message));
    printf("Child wrote: %s\n", shmem);

  } else {
    printf("Parent read: %s\n", shmem);
    sleep(1);
    printf("After 1s, parent read: %s\n", shmem);
  }
}

Without extra management layer

Process 1 creates a file and allocates memory on that with MAP_SHARED flag and appropriate protection level and length. This process can write some data in the allocated memory space.
Process 1 shares this file descriptor with Process 2 via a certain IPC method.
Process 2 receives this file descriptor and calls mmap on that. So the system returns the virtual address of the same memory allocation and based on the protection levels set by Process 1, Process 2 can read, write or execute the shared memory pages. These processes are responsible for explicitly deallocating memory, otherwise, it cannot be reused by another process in need of memory.

With an extra management layer In this case, another process acts as the manager of shared memory and exposes interface or methods to allocate or retrieve memory allocations. Let’s say there is a memory manager called XMAN and exposes APIs like this:

1
2
3
4
5
6
7
8
9


// Hypothetical XMAN APIS
struct Xman_allocation {
    void *addr; // address of allocation
    int fd;     // file descriptor
};

Xman_allocation Xman_allocate(size_t length, int prot);
Xman_allocation Xman_get(int fd);
void Xman_free(Xman_allocation allocation);

Process 1 could allocate a chunk of memory using Xman_allocate() and share the Xman_allocation.fd with another process via a certain IPC mechanism.
Process 2 could use Xman_get() to get the same allocation and act on it.
Any of these processes could use the Xman_free() to explicitly free the memory.

shm_open

shm_open() creates and opens a new, or opens an existing, POSIX shared memory object. A POSIX shared memory object is in effect a handle which can be used by unrelated processes to mmap the same region of shared memory. The shm_unlink() function performs the converse operation, removing an object previously created by shm_open().

1
2
3
4
5
6


#include <sys/mman.h>
#include <sys/stat.h> /* For mode constants */
#include <fcntl.h> /* For O_* constants */
int shm_open(const char *name, int oflag, mode_t mode);

int shm_unlink(const char *name);

example：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


// producer.cpp
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <sys/shm.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <unistd.h>
#include <sys/types.h>
int main()
{
  const int SIZE = 4096;
  const char *name = "OS";
  const char *message0= "Studying ";
  const char *message1= "Operating Systems ";
  const char *message2= "Is Fun!";

  int shm_fd;
  void *ptr;

  /* create the shared memory segment */
  shm_fd = shm_open(name, O_CREAT | O_RDWR, 0666);

  /* configure the size of the shared memory segment */
  ftruncate(shm_fd,SIZE);

  /* now map the shared memory segment in the address space of the process */
  ptr = mmap(0,SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, shm_fd, 0);
  if (ptr == MAP_FAILED) {
    printf("Map failed\n");
    return -1;
  }

  /**
   * Now write to the shared memory region.
   *
   * Note we must increment the value of ptr after each write.
   */
  memcpy(ptr, message0, strlen(message0));
  ptr += strlen(message0);
  memcpy(ptr, message1, strlen(message1));
  ptr += strlen(message1);
  memcpy(ptr, message2, strlen(message2));
  ptr += strlen(message2);
  return 0;
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


// consumer.cpp
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <sys/shm.h>
#include <sys/stat.h>
#include <sys/mman.h>

int main()
{
  const char *name = "OS";
  const int SIZE = 4096;

  int shm_fd;
  void *ptr;
  int i;

  /* open the shared memory segment */
  shm_fd = shm_open(name, O_RDONLY, 0666);
  if (shm_fd == -1) {
    printf("shared memory failed\n");
    exit(-1);
  }

  /* now map the shared memory segment in the address space of the process */
  ptr = mmap(0,SIZE, PROT_READ, MAP_SHARED, shm_fd, 0);
  if (ptr == MAP_FAILED) {
    printf("Map failed\n");
    exit(-1);
  }

  /* now read from the shared memory region */
  printf("%s",ptr);

  /* remove the shared memory segment */
  if (shm_unlink(name) == -1) {
    printf("Error removing %s\n",name);
    exit(-1);
  }

  return 0;
}

Advance

用生成文件的方式创建可用于共享的内存。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


/* Create a buffer. This is creating a temporary file and then
 * immediately unlinking it so we do not leave traces in the system. */
int create_buffer(int64_t size) {
  static char template[] = "/tmp/plasmaXXXXXX";
  char file_name[32];
  strncpy(file_name, template, 32);
  int fd = mkstemp(file_name);
  if (fd < 0)
    return -1;
  FILE *file = fdopen(fd, "a+");
  if (!file) {
    close(fd);
    return -1;
  }
  if (unlink(file_name) != 0) {
    LOG_ERR("unlink error");
    return -1;
  }
  if (ftruncate(fd, (off_t) size) != 0) {
    LOG_ERR("ftruncate error");
    return -1;
  }
  return fd;
}

Shared Memory

mmap

mmap definition

munmap definition

Sharing memory with mmap

Sharing between parent and child

Sharing between siblings

shm_open

Advance

Reference

相关文章：