Pingora provides extensive customizability to your network programming. For more information, read: the this article.

Problem: network failure

In shutdown, the upstream backend may not be able to respond. To deal with those kinds of situation, Pingora's ProxyHttp provides failover-related filters such as fail_to_connect and error_while_proxy. In this post, fail_to_connect will be investigated.

fail_to_connect

When fail_to_connect is invoked, pingora-proxy guarantees that nothing was sent to upstream backend server. Does that mean that it works even when processing from backend side takes more than configured connection_timeout value? Let's check this out.

Firstly, we need to boot up the server

#[tokio::main]
async fn main() {
    let host_port = args().nth(1).unwrap_or("0.0.0.0:44447".to_string());

    println!("Starting server...{}", host_port);
    let listener = tokio::net::TcpListener::bind(host_port).await.unwrap();

    axum::serve(listener, router()).await.unwrap();
}

fn router() -> Router {
    Router::new().route("/", get(|| async { "Hello, World!" }))
}

And then on the Pingora side, we need the following set-up.

// To count the number of tries
pub struct TryCounter {
    tries: usize,
}

#[async_trait]
impl ProxyHttp for LB {

    type CTX = TryCounter;
    fn new_ctx(&self) -> Self::CTX {
        MyCtx { tries: 0 }
    }
    
    async fn upstream_peer(
        &self,
        _session: &mut Session,
        ctx: &mut Self::CTX,
    ) -> Result<Box<HttpPeer>> {
        let addr = if ctx.tries < 1 {
            ("0.0.0.0", 44410)
        } else {
            ("0.0.0.0", 44447)
        };
        println!("found upstream peer...{:?}", addr);

        let mut peer = Box::new(HttpPeer::new(addr, false, "".to_string()));
        peer.options.connection_timeout = Some(Duration::from_millis(400));
        Ok(peer)
    }
    
    fn fail_to_connect(
        &self,
        _session: &mut Session,
        _peer: &HttpPeer,
        ctx: &mut Self::CTX,
        mut e: Box<Error>,
    ) -> Box<Error> {
        println!("fail_to_connect!");
        if ctx.tries > 0 {
            return e;
        }
        ctx.tries += 1;
        e.set_retry(true);
        e
    }
}

Things to understand - CTX

Context(CTX) is request-specific information, so each request owns a single CTX. Hoping around the filters, each filter can read and update CTX information and CTX is dropped at the end of request.

Now, with this set up, you run two upstream servers

#On one terminal
cargo run 0.0.0.0:44410

#On the other terminal
cargo run 0.0.0.0:44447

Now, you boot up the pingora proxy as well and make a request:

curl http://0.0.0.0:44444

# response
Hello, World!

Okay, now, you shut down the server using 44410 port:

curl http://0.0.0.0:44444

# response
Hello, World!

Then failover works, the request is routed to server holding 44447 port.

But then again, what if it just takes longer time than value set to HttpPeer?

Let's make a quick modification on upstream backend side:

fn router() -> Router {
    Router::new().route(
        "/",
        get(|| async {
		// Add sleep 
        tokio::time::sleep(tokio::time::Duration::from_secs(1)).await;
            "Hello, World!"
        }),
    )
}

And then you load it up.

cargo run 0.0.0.0:44410

Now, when you make a request again, it doesn't reroute the request to 44447 port.

What it means is that fail_to_connect is invoked only when there is an error in the process of establishing a connection.