HTTP server from TCP socket (Go)

I'm trying to create a Go TCP socket, bind it to a VRF interface , and set up an HTTP server on that specific interface. The VRF binding works correctly, but upon starting the HTTP server returns an error stating "Accept TCP 127.0.0.1:80: Accept: Invalid argument" . Am I right to assume that the socket is some kind of defect that I created wrong?

Below is a simplified version to reproduce the problem. The VRF part is commented out because it doesn't affect the actual problem, but I'm taking it here as someone I'm trying to avoid telling me to just use net.Listen instead of sockets. VRF needs to be bound first before you can use it, so net.Listen is not an unfortunate option.

package main

import (
    "fmt"
    "net"
    "net/http"
    "os"
    "syscall"
)

func main() {
    fd, err := syscall.Socket(syscall.AF_INET, syscall.SOCK_STREAM, syscall.IPPROTO_TCP)
    if err != nil {
        fmt.Printf("Error creating socket: %v", err)
        os.Exit(1)
    }

    // if err = syscall.SetsockoptString(fd, syscall.SOL_SOCKET, syscall.SO_BINDTODEVICE, "vrfiface"); err != nil {
    //  fmt.Printf("Error binding to vrf: %v", err)
    //  os.Exit(1)
    // }

    sockAddr := &syscall.SockaddrInet4{
        Port: 80,
        Addr: [4]byte{127, 0, 0, 1},
    }

    if err = syscall.Bind(fd, sockAddr); err != nil {
        fmt.Printf("Error binding to IP and port: %v", err)
        os.Exit(1)
    }

    file := os.NewFile(uintptr(fd), "socketfile")
    if file == nil {
        fmt.Println("Error creating file")
        os.Exit(1)
    }

    listener, err := net.FileListener(file)
    if err != nil {
        fmt.Printf("Error creating a listener: %v", err)
        os.Exit(1)
    }

    http.HandleFunc("/", TestServer)
    if err = http.Serve(listener, nil); err != nil {
        fmt.Printf("Error serving HTTP requests: %v", err)
    }
}

func TestServer(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Test, %s!", r.URL.Path[1:])
}

Any pointers to fix this would be greatly appreciated. Thanks!