turborepo / nextjs compat issue

[aviramha]

User reported that running next repo using turbopack doesn't work with mirrord

webpack --> mirrord exec -f .mirrord/mirrord.json -- next dev -p 4004
  turbopack --> mirrord exec -f .mirrord/mirrord.json -- next dev --turbopack -p 4004

We had this issue in the past and thought same workaround would work but when running from next cli you can't really add the workaround configuration.

I thought the fix would be in Vercel's Turborepo side but they didn't allow us to include mirrord env variables.

This leads me to the following idea - instead of saving settings inside env variables and trust it'd propagate between sub processes, we will use a const fd + memfd to store the data.

On mirrord config access -> check if our const 94838 FD is valid, if it is, check if it has magic in first 4 bytes, if it does, load encoded settings from there -> can repeat and iterate on FDs in case our FD is used already for other means -> if we can't find in fd fallback to env.

[linear]

MBE-1494

[Razz4780]

Using a const FD might be problematic. In some environments we might have a very low FD limit (256). This would require increasing the FD limit

[aviramha]

Using a const FD might be problematic. In some environments we might have a very low FD limit (256). This would require increasing the FD limit

Are you sure the limit also limits the potential number?

[aviramha]

Yup. Nice catch

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/resource.h>

int main() {
    struct rlimit rl;
    
    // Set max fd to 10
    rl.rlim_cur = 10;
    rl.rlim_max = 10;
    if (setrlimit(RLIMIT_NOFILE, &rl) == -1) {
        perror("setrlimit");
        exit(1);
    }
    
    // Open /dev/null
    int fd = open("/dev/null", O_RDWR);
    if (fd == -1) {
        perror("open");
        exit(1);
    }
    printf("Opened /dev/null, fd = %d\n", fd);
    
    // Try to duplicate to fd 4000
    int newfd = fcntl(fd, F_DUPFD, 4000);
    if (newfd == -1) {
        perror("fcntl F_DUPFD");
        printf("Failed (expected - fd 4000 exceeds limit of 10)\n");
    } else {
        printf("Duplicated to fd = %d\n", newfd);
        close(newfd);
    }
    
    close(fd);
    return 0;
}

./main
Opened /dev/null, fd = 3
fcntl F_DUPFD: Invalid argument
Failed (expected - fd 4000 exceeds limit of 10)

[aviramha]

Other ideas how we can propagate settings? I guess a file :( but then we can't distinguish which exact session. uuf

[aviramha]

Okay, 🐻 with me - iterate from the bottom the first 10 fds? mirrord loads very early so should catch one of the first ones.

[Razz4780]

Another idea - let's preload the layer via soft link

1. CLI extracts the layer to known location, as usual
2. CLI creates a tmp dir with a soft link to the layer, and the new file for env vars
3. Layer finds its tmp dir from the path in LD_PRELOAD

[aviramha]

I'd like to refrain having the binaries in temp path because we know it might trigger some AVs
You don't like my fds fun idea?

[iniw]

Wonder if this is related to COR-203 at all? I just started working on it yesterday :P

[aviramha]

Wonder if this is related to COR-203 at all? I just started working on it yesterday :P

Most likely not

[Razz4780]

I'd like to refrain having the binaries in temp path because we know it might trigger some AVs You don't like my fds fun idea?

I think layer is already being loaded from tmp tho 🤔

The fd idea is cool, why not ^^

[Razz4780]

I suggested another approach, because with the fd solution we'd still need both methods (env and fd), to handle mirrord ext flow

[aviramha]

I think layer is already being loaded from tmp tho 🤔

We're kinda moving away from there slowly as more Enterprises ask us to have more static location

I suggested another approach, because with the fd solution we'd still need both methods (env and fd), to handle mirrord ext flow

Yeah that makes sense, I'm assuming since we're always meddling with the execution it's very unlikely env will drop in first process we get loaded to

[aviramha]

@Razz4780 perhaps we should always handle it in our execve hooks?