Over the past 50 years, changes in the intersectoral water use in the Rufiji Basin have been enormous. A growing human population, migration and increasing demands in the basin have culminated this change. The basin, however, still lack an appropriate integrated management approach. This has resulted into inter-institutional conflicts, ineffectiveness, gaps in management imperatives and duplication of efforts. This paper reviews the existing institutional linkages identifies the gap and proposes an appropriate institutional framework which involve questions of institutional arrangements and the assignment of responsibilities among various levels of development, ensures stakeholders participation, accommodates adaptive change and remain self sustainable. The basic argument of this paper is that water management issue is both a question of developing stakeholders' participation and transferring state's competence to water user associations. Such an endeavour requires a complete and complex institutional framework, which would define clearly the role and rule of each stakeholder in water resource management. The paper further argues that; in Tanzania, the institutions that are involved in water management are loosely connected and lack basic coordination and are often at the periphery of the water management agenda -- divorced from the water management programs; the predominance of isolated institutions locked up in narrowly defined activities with no interactive learning process will continue to hamper national aspirations to manage water; and that to change this situation will require innovative reforms in national institutions and institutional learning.