The most talked about line is about the rubber end of a pencil. This one is in wikiquote:
Bank manager: People make mistakes.
Fleabag: It's why they put rubbers on the ends of pencils.
Bank manager: Is that a joke?
Fleabag: I don't know.
Fleabag is in fact referencing a joke that is not posted in wikiquote, at least not yet:
Boo: OK, an 11-year-old boy was put in juvenile prison for repeatedly sticking rubber ended pencils up his school hamster's arsehole.
Boo: Yeah. Why would they do that?
Fleabag: Apparently he liked it when their eyes popped out.
Boo: No, why would they send him away? He needs help. ... They shouldn't have just locked him up. ... Yeah, but he's obviously not happy, happy people wouldn't do things like that.
Fleabag: Fair point.
Boo: And anyway, that's the very reason why they put rubbers on the end of pencils.
Fleabag: To f*ck hamsters?
Boo: No, because people make mistakes.